2023-12-05T22:06:44.134 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-05T22:06:44.134 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989 --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} --owner scheduled_dis@teuthology --name dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi --verbose -- /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/orig.config.yaml 2023-12-05T22:06:44.179 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-05T22:06:44.297 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: idryomov@gmail.com first_in_suite: false job_id: '7478989' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 4a50e96e7d9d85da28ef008617cb8342574bfd1b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 4a50e96e7d9d85da28ef008617cb8342574bfd1b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: 4a50e96e7d9d85da28ef008617cb8342574bfd1b owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2777 sha1: 4a50e96e7d9d85da28ef008617cb8342574bfd1b sleep_before_teardown: 0 subset: 17/99 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 4a50e96e7d9d85da28ef008617cb8342574bfd1b targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKqroaZT7HdYX62YLD5DEPqGdvn/11IHy8kmg0TszQ4KFHbms/zi4XIFQ/5ysjy8atJOWH5R6XjS7bmPmFzLnsc= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWBgB238hKlGR5M5K13I6ZaYPHPvcklmz6GAjCQypqGTttuc6O4NwhdgZf/28VITHQriAuy2J0LPslmczqnfYo= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-05_21:30:10 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-05T22:06:44.298 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa; will attempt to use it 2023-12-05T22:06:44.298 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks 2023-12-05T22:06:44.299 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-05T22:06:44.300 INFO:teuthology.task.internal:Checking packages... 2023-12-05T22:06:44.323 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4a50e96e7d9d85da28ef008617cb8342574bfd1b' 2023-12-05T22:06:44.323 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-05T22:06:44.323 INFO:teuthology.packaging:ref: None 2023-12-05T22:06:44.323 INFO:teuthology.packaging:tag: None 2023-12-05T22:06:44.323 INFO:teuthology.packaging:branch: wip-dis-testing 2023-12-05T22:06:44.324 INFO:teuthology.packaging:sha1: 4a50e96e7d9d85da28ef008617cb8342574bfd1b 2023-12-05T22:06:44.324 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2023-12-05T22:06:44.489 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:06:44.490 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-05T22:06:44.495 INFO:teuthology.task.internal:no buildpackages task found 2023-12-05T22:06:44.495 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-05T22:06:44.509 INFO:teuthology.task.internal:Saving configuration 2023-12-05T22:06:44.526 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-05T22:06:44.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-05T22:06:44.558 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-05 22:01:04.971197', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHW0XxlXzz6f19xpz42vQtcKXIwKVQdO+wsZkK+/4j/6'} 2023-12-05T22:06:44.581 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-05 22:01:04.972647', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOeOTyhZ1X5xmDzzymazUEqHTxvPSzKZRKZHxSgks6mR'} 2023-12-05T22:06:44.581 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-05T22:06:44.586 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-05T22:06:44.586 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-05T22:06:44.586 INFO:teuthology.run_tasks:Running task console_log... 2023-12-05T22:06:44.666 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcc2a039f70>, signals=[15]) 2023-12-05T22:06:44.667 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-05T22:06:44.672 INFO:teuthology.task.internal:Opening connections... 2023-12-05T22:06:44.673 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T22:06:44.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:06:44.753 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T22:06:44.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:06:44.829 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-05T22:06:44.834 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2023-12-05T22:06:44.839 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2023-12-05T22:06:44.839 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Ubuntu" 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="22.04" 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_CODENAME=jammy 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:ID=ubuntu 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE=debian 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-05T22:06:44.885 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-05T22:06:44.886 INFO:teuthology.orchestra.run.smithi062.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-05T22:06:44.886 INFO:teuthology.orchestra.run.smithi062.stdout:UBUNTU_CODENAME=jammy 2023-12-05T22:06:44.887 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2023-12-05T22:06:44.912 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2023-12-05T22:06:44.917 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2023-12-05T22:06:44.918 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2023-12-05T22:06:44.964 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-05T22:06:44.964 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Ubuntu" 2023-12-05T22:06:44.964 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="22.04" 2023-12-05T22:06:44.964 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_CODENAME=jammy 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:ID=ubuntu 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE=debian 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-05T22:06:44.965 INFO:teuthology.orchestra.run.smithi190.stdout:UBUNTU_CODENAME=jammy 2023-12-05T22:06:44.965 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2023-12-05T22:06:44.987 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-05T22:06:44.996 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-05T22:06:45.023 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-05T22:06:45.024 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2023-12-05T22:06:45.027 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2023-12-05T22:06:45.032 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-05T22:06:45.038 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-05T22:06:45.038 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2023-12-05T22:06:45.075 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2023-12-05T22:06:45.088 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-05T22:06:45.141 INFO:teuthology.run_tasks:Running task kernel... 2023-12-05T22:06:45.155 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-05T22:06:45.156 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-05T22:06:45.156 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-05T22:06:45.156 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-05T22:06:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:06:45.161 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2023-12-05T22:06:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:06:45.162 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-12-05T22:06:45.209 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-56-generic 2023-12-05T22:06:45.209 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.15.0-56-generic 2023-12-05T22:06:45.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2023-12-05T22:06:45.210 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-56-generic 2023-12-05T22:06:45.211 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.15.0-56-generic 2023-12-05T22:06:45.211 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2023-12-05T22:06:45.328 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2023-12-05T22:06:45.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2023-12-05T22:06:45.459 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-05T22:06:45.474 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-05T22:06:45.576 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-05T22:06:45.589 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-05T22:06:45.641 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [374 kB] 2023-12-05T22:06:45.658 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-05T22:06:45.667 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,012 kB] 2023-12-05T22:06:45.671 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-05T22:06:45.704 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,012 kB] 2023-12-05T22:06:45.734 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [194 kB] 2023-12-05T22:06:45.738 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-05T22:06:45.738 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-05T22:06:45.738 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,179 kB] 2023-12-05T22:06:45.752 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [191 kB] 2023-12-05T22:06:45.754 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-05T22:06:45.754 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [811 kB] 2023-12-05T22:06:45.754 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [374 kB] 2023-12-05T22:06:45.759 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [194 kB] 2023-12-05T22:06:45.760 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [576 kB] 2023-12-05T22:06:45.761 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-05T22:06:45.761 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,179 kB] 2023-12-05T22:06:45.764 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [151 kB] 2023-12-05T22:06:45.766 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-05T22:06:45.779 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-05T22:06:45.780 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [191 kB] 2023-12-05T22:06:45.782 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-05T22:06:45.782 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [576 kB] 2023-12-05T22:06:45.787 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-05T22:06:45.787 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-05T22:06:45.787 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-05T22:06:45.787 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-05T22:06:45.788 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [811 kB] 2023-12-05T22:06:45.793 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [151 kB] 2023-12-05T22:06:45.794 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-05T22:06:45.795 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-05T22:06:45.801 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-05T22:06:45.812 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-05T22:06:45.812 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-05T22:06:46.023 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-05T22:06:46.033 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-05T22:06:46.159 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [538 kB] 2023-12-05T22:06:46.172 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,223 kB] 2023-12-05T22:06:46.346 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,223 kB] 2023-12-05T22:06:46.454 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [538 kB] 2023-12-05T22:06:46.479 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [254 kB] 2023-12-05T22:06:46.491 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [254 kB] 2023-12-05T22:06:46.496 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-05T22:06:46.497 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-05T22:06:46.498 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,199 kB] 2023-12-05T22:06:46.509 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-05T22:06:46.509 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,199 kB] 2023-12-05T22:06:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [194 kB] 2023-12-05T22:06:46.559 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-05T22:06:46.564 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-05T22:06:46.565 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [194 kB] 2023-12-05T22:06:46.572 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-05T22:06:46.723 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,017 kB] 2023-12-05T22:06:46.737 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,017 kB] 2023-12-05T22:06:47.350 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [674 kB] 2023-12-05T22:06:47.369 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [674 kB] 2023-12-05T22:06:47.388 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [225 kB] 2023-12-05T22:06:47.400 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-05T22:06:47.402 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-05T22:06:47.402 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-05T22:06:47.404 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-05T22:06:47.405 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-05T22:06:47.405 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-05T22:06:47.407 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [225 kB] 2023-12-05T22:06:47.420 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-05T22:06:47.421 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-05T22:06:47.424 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-05T22:06:47.424 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-05T22:06:47.425 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-05T22:06:47.425 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-05T22:06:47.432 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-05T22:06:47.435 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-05T22:06:47.451 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-05T22:06:47.454 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-05T22:06:47.600 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-05T22:06:47.618 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-05T22:06:47.682 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-05T22:06:47.701 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-05T22:06:47.793 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-05T22:06:47.817 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-05T22:06:47.848 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-05T22:06:47.866 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-05T22:06:47.882 INFO:teuthology.orchestra.run.smithi062.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-05T22:06:47.901 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-05T22:06:52.263 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 10.6 MB in 2s (4,245 kB/s) 2023-12-05T22:06:52.569 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 10.6 MB in 2s (4,253 kB/s) 2023-12-05T22:06:53.627 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-12-05T22:06:53.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-05T22:06:53.705 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-12-05T22:06:53.803 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-12-05T22:06:53.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-05T22:06:53.883 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-12-05T22:06:53.893 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2023-12-05T22:06:53.894 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2023-12-05T22:06:54.047 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2023-12-05T22:06:54.047 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-05T22:06:54.047 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-05T22:06:54.047 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-05T22:06:54.048 INFO:teuthology.orchestra.run.smithi190.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-05T22:06:54.048 INFO:teuthology.orchestra.run.smithi190.stdout: python2.7-minimal 2023-12-05T22:06:54.048 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-05T22:06:54.048 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2023-12-05T22:06:54.048 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-05T22:06:54.049 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-generic linux-image-5.15.0-89-generic 2023-12-05T22:06:54.049 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-5.15.0-89-generic linux-modules-extra-5.15.0-89-generic 2023-12-05T22:06:54.050 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2023-12-05T22:06:54.050 INFO:teuthology.orchestra.run.smithi190.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-05T22:06:54.050 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2023-12-05T22:06:54.050 INFO:teuthology.orchestra.run.smithi190.stdout: thermald 2023-12-05T22:06:54.079 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2023-12-05T22:06:54.080 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2023-12-05T22:06:54.104 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2023-12-05T22:06:54.104 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-05T22:06:54.104 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.15.0-89-generic linux-modules-5.15.0-89-generic 2023-12-05T22:06:54.105 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-extra-5.15.0-89-generic 2023-12-05T22:06:54.105 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages will be upgraded: 2023-12-05T22:06:54.106 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-05T22:06:54.182 INFO:teuthology.orchestra.run.smithi190.stdout:3 upgraded, 5 newly installed, 0 to remove and 274 not upgraded. 2023-12-05T22:06:54.182 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 113 MB of archives. 2023-12-05T22:06:54.182 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-05T22:06:54.182 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-89-generic amd64 5.15.0-89.99 [22.4 MB] 2023-12-05T22:06:54.249 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2023-12-05T22:06:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-05T22:06:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-05T22:06:54.249 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-05T22:06:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-05T22:06:54.250 INFO:teuthology.orchestra.run.smithi062.stdout: python2.7-minimal 2023-12-05T22:06:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-05T22:06:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2023-12-05T22:06:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-05T22:06:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-generic linux-image-5.15.0-89-generic 2023-12-05T22:06:54.251 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-5.15.0-89-generic linux-modules-extra-5.15.0-89-generic 2023-12-05T22:06:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2023-12-05T22:06:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-05T22:06:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2023-12-05T22:06:54.253 INFO:teuthology.orchestra.run.smithi062.stdout: thermald 2023-12-05T22:06:54.307 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2023-12-05T22:06:54.309 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-12-05T22:06:54.309 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.15.0-89-generic linux-modules-5.15.0-89-generic 2023-12-05T22:06:54.309 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-extra-5.15.0-89-generic 2023-12-05T22:06:54.310 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages will be upgraded: 2023-12-05T22:06:54.310 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-05T22:06:54.491 INFO:teuthology.orchestra.run.smithi062.stdout:3 upgraded, 5 newly installed, 0 to remove and 274 not upgraded. 2023-12-05T22:06:54.491 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 113 MB of archives. 2023-12-05T22:06:54.491 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-05T22:06:54.491 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-89-generic amd64 5.15.0-89.99 [22.4 MB] 2023-12-05T22:06:54.777 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-89-generic amd64 5.15.0-89.99 [11.5 MB] 2023-12-05T22:06:54.926 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-89-generic amd64 5.15.0-89.99 [63.9 MB] 2023-12-05T22:06:55.911 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-89-generic amd64 5.15.0-89.99 [11.5 MB] 2023-12-05T22:06:56.099 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.89.86 [1,698 B] 2023-12-05T22:06:56.100 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.89.86 [2,492 B] 2023-12-05T22:06:56.100 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89 all 5.15.0-89.99 [12.3 MB] 2023-12-05T22:06:56.261 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-89-generic amd64 5.15.0-89.99 [63.9 MB] 2023-12-05T22:06:56.296 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89-generic amd64 5.15.0-89.99 [2,885 kB] 2023-12-05T22:06:56.347 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.89.86 [2,336 B] 2023-12-05T22:06:56.743 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 113 MB in 2s (50.6 MB/s) 2023-12-05T22:06:56.902 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-5.15.0-89-generic. 2023-12-05T22:06:58.464 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.89.86 [1,698 B] 2023-12-05T22:06:58.465 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.89.86 [2,492 B] 2023-12-05T22:06:58.465 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89 all 5.15.0-89.99 [12.3 MB] 2023-12-05T22:06:58.885 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-05T22:06:58.889 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../0-linux-modules-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:06:58.982 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:06:59.283 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89-generic amd64 5.15.0-89.99 [2,885 kB] 2023-12-05T22:06:59.410 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.89.86 [2,336 B] 2023-12-05T22:06:59.814 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 113 MB in 5s (22.2 MB/s) 2023-12-05T22:06:59.977 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-5.15.0-89-generic. 2023-12-05T22:07:01.016 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-image-5.15.0-89-generic. 2023-12-05T22:07:01.037 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../1-linux-image-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:07:01.125 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:01.577 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-89-generic. 2023-12-05T22:07:01.598 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:07:01.652 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:01.938 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-05T22:07:01.943 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../0-linux-modules-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:07:02.025 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:04.359 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-image-5.15.0-89-generic. 2023-12-05T22:07:04.381 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../1-linux-image-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:07:04.470 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:04.954 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-89-generic. 2023-12-05T22:07:04.975 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:07:05.012 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:08.216 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../3-linux-generic_5.15.0.89.86_amd64.deb ... 2023-12-05T22:07:08.326 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-05T22:07:08.659 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.89.86_amd64.deb ... 2023-12-05T22:07:08.788 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-05T22:07:09.074 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-89. 2023-12-05T22:07:09.096 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../5-linux-headers-5.15.0-89_5.15.0-89.99_all.deb ... 2023-12-05T22:07:09.141 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-05T22:07:11.282 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../3-linux-generic_5.15.0.89.86_amd64.deb ... 2023-12-05T22:07:11.417 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-05T22:07:11.751 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.89.86_amd64.deb ... 2023-12-05T22:07:11.862 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-05T22:07:12.123 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-89. 2023-12-05T22:07:12.134 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../5-linux-headers-5.15.0-89_5.15.0-89.99_all.deb ... 2023-12-05T22:07:12.173 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-05T22:07:15.158 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-89-generic. 2023-12-05T22:07:15.194 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../6-linux-headers-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:07:15.233 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:17.822 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.89.86_amd64.deb ... 2023-12-05T22:07:17.944 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-05T22:07:17.957 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-89-generic. 2023-12-05T22:07:17.994 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../6-linux-headers-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-12-05T22:07:18.056 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:18.276 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-05T22:07:18.374 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:18.475 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-generic (5.15.0.89.86) ... 2023-12-05T22:07:18.605 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:19.917 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-05T22:07:19.918 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-05T22:07:19.918 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-89-generic 2023-12-05T22:07:19.918 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:20.097 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:20.786 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.89.86_amd64.deb ... 2023-12-05T22:07:20.910 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-12-05T22:07:21.670 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-12-05T22:07:21.799 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:21.799 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-generic (5.15.0.89.86) ... 2023-12-05T22:07:21.925 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-generic (5.15.0.89.86) ... 2023-12-05T22:07:21.925 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-generic (5.15.0.89.86) ... 2023-12-05T22:07:22.030 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:22.051 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:23.374 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-05T22:07:23.374 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-05T22:07:23.374 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-89-generic 2023-12-05T22:07:23.374 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:23.420 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:23.483 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-05T22:07:23.483 INFO:teuthology.orchestra.run.smithi190.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:23.539 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:24.990 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-generic (5.15.0.89.86) ... 2023-12-05T22:07:25.118 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-generic (5.15.0.89.86) ... 2023-12-05T22:07:25.250 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:26.715 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-12-05T22:07:27.427 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-05T22:07:27.427 INFO:teuthology.orchestra.run.smithi062.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:37.290 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-05T22:07:37.290 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub' 2023-12-05T22:07:37.309 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-05T22:07:37.336 INFO:teuthology.orchestra.run.smithi190.stdout:Generating grub configuration file ... 2023-12-05T22:07:37.607 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-05T22:07:37.631 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:37.918 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-05T22:07:37.924 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-05T22:07:37.993 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-05T22:07:37.998 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-05T22:07:38.166 INFO:teuthology.orchestra.run.smithi190.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-05T22:07:38.167 INFO:teuthology.orchestra.run.smithi190.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-05T22:07:38.167 INFO:teuthology.orchestra.run.smithi190.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-05T22:07:38.226 INFO:teuthology.orchestra.run.smithi190.stdout:done 2023-12-05T22:07:38.664 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2023-12-05T22:07:38.688 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 20 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.89.86 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2023-12-05T22:07:38.689 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2023-12-05T22:07:38.690 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2023-12-05T22:07:38.690 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-89-generic 2023-12-05T22:07:38.690 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-05T22:07:38.690 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-05T22:07:38.690 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-05T22:07:38.690 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2023-12-05T22:07:38.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2023-12-05T22:07:38.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2023-12-05T22:07:39.133 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-05T22:07:39.133 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-05T22:07:39.197 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-05T22:07:39.279 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-05T22:07:40.431 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-12-05T22:07:40.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-05T22:07:40.500 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-12-05T22:07:40.674 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2023-12-05T22:07:40.674 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2023-12-05T22:07:40.821 INFO:teuthology.orchestra.run.smithi190.stdout:linux-image-generic is already the newest version (5.15.0.89.86). 2023-12-05T22:07:40.821 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2023-12-05T22:07:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-05T22:07:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-05T22:07:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-05T22:07:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-05T22:07:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-05T22:07:40.822 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-05T22:07:40.859 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 0 newly installed, 0 to remove and 274 not upgraded. 2023-12-05T22:07:40.861 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2023-12-05T22:07:40.915 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 20 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.89.86 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2023-12-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2023-12-05T22:07:40.917 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2023-12-05T22:07:40.964 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.AKEW8ibvWu 2023-12-05T22:07:40.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub/grub.cfg /tmp/tmp.AKEW8ibvWu 2023-12-05T22:07:41.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.AKEW8ibvWu 2023-12-05T22:07:41.128 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.AKEW8ibvWu is 10KB 2023-12-05T22:07:41.142 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.AKEW8ibvWu 2023-12-05T22:07:41.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-05T22:07:41.203 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:07:41.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-05T22:07:41.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-05T22:07:41.271 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-89-generic 2023-12-05T22:07:41.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo update-grub 2023-12-05T22:07:41.665 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-05T22:07:41.665 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub' 2023-12-05T22:07:41.685 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-05T22:07:41.708 INFO:teuthology.orchestra.run.smithi062.stdout:Generating grub configuration file ... 2023-12-05T22:07:41.959 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-05T22:07:41.980 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:42.255 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-05T22:07:42.261 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-05T22:07:42.339 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-05T22:07:42.345 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-05T22:07:42.423 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub' 2023-12-05T22:07:42.424 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-05T22:07:42.427 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2023-12-05T22:07:42.494 INFO:teuthology.orchestra.run.smithi062.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-05T22:07:42.494 INFO:teuthology.orchestra.run.smithi062.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-05T22:07:42.494 INFO:teuthology.orchestra.run.smithi062.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-05T22:07:42.541 INFO:teuthology.orchestra.run.smithi062.stdout:done 2023-12-05T22:07:42.606 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-05T22:07:42.612 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:42.835 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-05T22:07:42.841 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-05T22:07:42.883 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-05T22:07:42.888 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-05T22:07:43.017 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-05T22:07:43.017 INFO:teuthology.orchestra.run.smithi190.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-05T22:07:43.017 INFO:teuthology.orchestra.run.smithi190.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-05T22:07:43.019 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2023-12-05T22:07:43.039 INFO:teuthology.orchestra.run.smithi190.stderr:done 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 20 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.89.86 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-05T22:07:43.045 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-05T22:07:43.046 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2023-12-05T22:07:43.046 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2023-12-05T22:07:43.046 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2023-12-05T22:07:43.046 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2023-12-05T22:07:43.046 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-89-generic 2023-12-05T22:07:43.046 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-05T22:07:43.046 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-05T22:07:43.046 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-05T22:07:43.046 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2023-12-05T22:07:43.046 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2023-12-05T22:07:43.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2023-12-05T22:07:43.272 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2023-12-05T22:07:43.398 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-05T22:07:43.525 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-05T22:07:43.600 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-05T22:07:43.681 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-05T22:07:44.792 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-12-05T22:07:44.809 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-05T22:07:44.871 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-12-05T22:07:45.063 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2023-12-05T22:07:45.063 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2023-12-05T22:07:45.232 INFO:teuthology.orchestra.run.smithi062.stdout:linux-image-generic is already the newest version (5.15.0.89.86). 2023-12-05T22:07:45.232 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2023-12-05T22:07:45.232 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-05T22:07:45.232 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-05T22:07:45.232 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-05T22:07:45.233 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-05T22:07:45.233 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-05T22:07:45.233 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-05T22:07:45.275 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 0 newly installed, 0 to remove and 274 not upgraded. 2023-12-05T22:07:45.277 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 20 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2023-12-05T22:07:45.290 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2023-12-05T22:07:45.291 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.89.86 2023-12-05T22:07:45.291 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-05T22:07:45.291 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-05T22:07:45.291 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2023-12-05T22:07:45.291 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2023-12-05T22:07:45.291 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2023-12-05T22:07:45.291 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2023-12-05T22:07:45.291 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2023-12-05T22:07:45.337 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.D1dN5lWjUv 2023-12-05T22:07:45.337 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub/grub.cfg /tmp/tmp.D1dN5lWjUv 2023-12-05T22:07:45.392 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.D1dN5lWjUv 2023-12-05T22:07:45.518 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.D1dN5lWjUv is 10KB 2023-12-05T22:07:45.531 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.D1dN5lWjUv 2023-12-05T22:07:45.537 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-05T22:07:45.594 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:07:45.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-05T22:07:45.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-05T22:07:45.660 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-89-generic 2023-12-05T22:07:45.660 DEBUG:teuthology.orchestra.run.smithi062:> sudo update-grub 2023-12-05T22:07:46.785 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub' 2023-12-05T22:07:46.786 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-05T22:07:46.788 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2023-12-05T22:07:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-12-05T22:07:46.972 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-12-05T22:07:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-05T22:07:47.193 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-05T22:07:47.234 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-05T22:07:47.239 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-05T22:07:47.378 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-05T22:07:47.378 INFO:teuthology.orchestra.run.smithi062.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-05T22:07:47.378 INFO:teuthology.orchestra.run.smithi062.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-05T22:07:47.390 INFO:teuthology.orchestra.run.smithi062.stderr:done 2023-12-05T22:07:47.415 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2023-12-05T22:08:13.074 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-05T22:08:13.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-12-05T22:08:13.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:08:17.423 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-05T22:08:17.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-12-05T22:08:17.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:08:31.587 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-12-05T22:08:35.938 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2023-12-05T22:08:40.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-12-05T22:08:40.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:08:43.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-12-05T22:08:44.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-12-05T22:08:44.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:08:48.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2023-12-05T22:08:55.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-12-05T22:08:55.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:09:00.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-12-05T22:09:00.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:09:14.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-12-05T22:09:18.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2023-12-05T22:09:29.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-12-05T22:09:29.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:09:29.505 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-12-05T22:09:30.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-12-05T22:09:30.228 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-89-generic"... 2023-12-05T22:09:30.228 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-12-05T22:09:30.274 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-89-generic 2023-12-05T22:09:30.274 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-89-generic vs 5.15.0-89-generic 2023-12-05T22:09:30.274 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-05T22:09:30.275 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-05T22:09:31.275 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-05T22:09:31.276 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-05T22:09:31.418 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2023-12-05T22:09:31.460 DEBUG:teuthology.parallel:result is None 2023-12-05T22:09:33.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-12-05T22:09:33.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:09:33.815 DEBUG:teuthology.orchestra.run.smithi062:> true 2023-12-05T22:09:34.769 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-12-05T22:09:34.770 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-89-generic"... 2023-12-05T22:09:34.770 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2023-12-05T22:09:34.816 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-89-generic 2023-12-05T22:09:34.816 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-89-generic vs 5.15.0-89-generic 2023-12-05T22:09:34.816 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-05T22:09:34.816 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-05T22:09:35.817 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-05T22:09:35.817 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-05T22:09:35.913 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2023-12-05T22:09:35.950 DEBUG:teuthology.parallel:result is None 2023-12-05T22:09:35.950 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-05T22:09:35.957 INFO:teuthology.task.internal:Creating test directory... 2023-12-05T22:09:35.957 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-05T22:09:35.959 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-05T22:09:35.966 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-05T22:09:35.977 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-05T22:09:35.990 INFO:teuthology.task.internal:Creating archive directory... 2023-12-05T22:09:35.990 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-05T22:09:36.009 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-05T22:09:36.031 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-05T22:09:36.038 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-05T22:09:36.038 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-05T22:09:36.065 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-05T22:09:36.082 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-05T22:09:36.085 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-05T22:09:36.091 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-05T22:09:36.093 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-05T22:09:36.095 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-05T22:09:36.102 INFO:teuthology.task.internal:Configuring sudo... 2023-12-05T22:09:36.102 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-05T22:09:36.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-05T22:09:36.150 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-05T22:09:36.158 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-05T22:09:36.158 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-05T22:09:36.193 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-05T22:09:36.199 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-05T22:09:36.245 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-05T22:09:36.292 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:09:36.293 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-05T22:09:36.367 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-05T22:09:36.373 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-05T22:09:36.418 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:09:36.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-05T22:09:36.486 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2023-12-05T22:09:36.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2023-12-05T22:09:36.585 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-05T22:09:36.592 INFO:teuthology.task.internal:Starting timer... 2023-12-05T22:09:36.592 INFO:teuthology.run_tasks:Running task pcp... 2023-12-05T22:09:36.601 INFO:teuthology.run_tasks:Running task selinux... 2023-12-05T22:09:36.609 DEBUG:teuthology.task.selinux:Excluding smithi062: OS 'ubuntu' does not support SELinux 2023-12-05T22:09:36.609 DEBUG:teuthology.task.selinux:Excluding smithi190: OS 'ubuntu' does not support SELinux 2023-12-05T22:09:36.609 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-05T22:09:36.609 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-05T22:09:36.609 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-05T22:09:36.609 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-05T22:09:36.618 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-05T22:09:36.618 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-05T22:09:36.636 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-05T22:09:36.637 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-05T22:17:28.390 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2023-12-05T22:17:28.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-12-05T22:17:28.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:17:28.476 DEBUG:teuthology.orchestra.run.smithi062:> true 2023-12-05T22:17:28.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-12-05T22:17:28.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-12-05T22:17:28.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-05T22:17:28.655 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-12-05T22:17:28.758 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-12-05T22:17:28.758 INFO:teuthology.run_tasks:Running task clock... 2023-12-05T22:17:28.769 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-05T22:17:28.769 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-05T22:17:28.770 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-05T22:17:28.772 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-05T22:17:28.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-05T22:17:28.800 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Command line: ntpd -gq 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: ---------------------------------------------------- 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: ntp-4 is maintained by Network Time Foundation, 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: corporation. Support and training for ntp-4 are 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: available at https://www.nwtime.org/support 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: ---------------------------------------------------- 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: proto: precision = 0.050 usec (-24) 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: basedate set to 2022-02-04 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: gps base set to 2022-02-06 (week 2196) 2023-12-05T22:17:28.801 INFO:teuthology.orchestra.run.smithi062.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-05T22:17:28.802 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-05T22:17:28.802 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: restrict ::: KOD does nothing without LIMITED. 2023-12-05T22:17:28.802 INFO:teuthology.orchestra.run.smithi062.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-05T22:17:28.803 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Listen and drop on 0 v6wildcard [::]:123 2023-12-05T22:17:28.803 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-05T22:17:28.803 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Listen normally on 2 lo 127.0.0.1:123 2023-12-05T22:17:28.803 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Listen normally on 3 enp3s0f1 172.21.15.62:123 2023-12-05T22:17:28.803 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Listen normally on 4 lo [::1]:123 2023-12-05T22:17:28.803 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:febb:d1c3%5]:123 2023-12-05T22:17:28.804 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:28 ntpd[28009]: Listening on routing socket on fd #22 for interface updates 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Command line: ntpd -gq 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: ---------------------------------------------------- 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: ntp-4 is maintained by Network Time Foundation, 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: corporation. Support and training for ntp-4 are 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: available at https://www.nwtime.org/support 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: ---------------------------------------------------- 2023-12-05T22:17:28.833 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: proto: precision = 0.063 usec (-24) 2023-12-05T22:17:28.834 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: basedate set to 2022-02-04 2023-12-05T22:17:28.834 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: gps base set to 2022-02-06 (week 2196) 2023-12-05T22:17:28.834 INFO:teuthology.orchestra.run.smithi190.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-05T22:17:28.835 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-05T22:17:28.835 INFO:teuthology.orchestra.run.smithi190.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-05T22:17:28.836 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: restrict ::: KOD does nothing without LIMITED. 2023-12-05T22:17:28.836 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Listen and drop on 0 v6wildcard [::]:123 2023-12-05T22:17:28.836 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-05T22:17:28.836 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Listen normally on 2 lo 127.0.0.1:123 2023-12-05T22:17:28.836 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Listen normally on 3 enp3s0f1 172.21.15.190:123 2023-12-05T22:17:28.836 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Listen normally on 4 lo [::1]:123 2023-12-05T22:17:28.836 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7763%5]:123 2023-12-05T22:17:28.837 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:28 ntpd[28015]: Listening on routing socket on fd #22 for interface updates 2023-12-05T22:17:29.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:29 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:29.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:29 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:29.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:29 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:29.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:29 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:29.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:29 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:29.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:29 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:29.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:29 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:29.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:29 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:29.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:29 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:29.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:29 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:29.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:29 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:29.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:29 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:30.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:30 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:30.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:30 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:30.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:30 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:30.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:30 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:30.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:30 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:30.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:30 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:30.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:30 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:31.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:31.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:31.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:31.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:31.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:31.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:31.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:31.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:31.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:31.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:31.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:31.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:31 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:31.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:31 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:31.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:31 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:31.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:31 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:31.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:31 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:31.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:31 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:31.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:31 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:32.800 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:32 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:32.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:32 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:32.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:32 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:32.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:32 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:32.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:32 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:32.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:32 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:32.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:32.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:32.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:32.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:32.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:32.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:32.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:32.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:32 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:33.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:33.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:33.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:33.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:33 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:33.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:33.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:33 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:34.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:34 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:34.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:34 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:34.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:34 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:34.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:34 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:34.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:34 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:34.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:34 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:34.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:34 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:34.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:34 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:34.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:34 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:34.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:34 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:34.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:34 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:34.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:34 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:35.801 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:35.802 INFO:teuthology.orchestra.run.smithi062.stdout: 5 Dec 22:17:35 ntpd[28009]: ntpd: time slew -0.000634 s 2023-12-05T22:17:35.802 INFO:teuthology.orchestra.run.smithi062.stdout:ntpd: time slew -0.000634s 2023-12-05T22:17:35.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-05T22:17:35.802 INFO:teuthology.orchestra.run.smithi062.stderr: 5 Dec 22:17:35 ntpd[28009]: can't open /var/log/ntpstats/loopstats.20231205: Permission denied 2023-12-05T22:17:35.833 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: can't open /var/log/ntpstats/rawstats.20231205: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: can't open /var/log/ntpstats/peerstats.20231205: Permission denied 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stdout: 5 Dec 22:17:35 ntpd[28015]: ntpd: time slew -0.001873 s 2023-12-05T22:17:35.834 INFO:teuthology.orchestra.run.smithi190.stdout:ntpd: time slew -0.001873s 2023-12-05T22:17:35.835 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-05T22:17:35.835 INFO:teuthology.orchestra.run.smithi190.stderr: 5 Dec 22:17:35 ntpd[28015]: can't open /var/log/ntpstats/loopstats.20231205: Permission denied 2023-12-05T22:17:35.888 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2023-12-05T22:17:35.888 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2023-12-05T22:17:35.888 INFO:teuthology.orchestra.run.smithi062.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.889 INFO:teuthology.orchestra.run.smithi062.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.889 INFO:teuthology.orchestra.run.smithi062.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.889 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.908 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2023-12-05T22:17:35.909 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2023-12-05T22:17:35.909 INFO:teuthology.orchestra.run.smithi190.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.909 INFO:teuthology.orchestra.run.smithi190.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.909 INFO:teuthology.orchestra.run.smithi190.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.909 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-05T22:17:35.910 INFO:teuthology.run_tasks:Running task install... 2023-12-05T22:17:35.921 DEBUG:teuthology.task.install:project ceph 2023-12-05T22:17:35.921 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}} 2023-12-05T22:17:35.921 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'} 2023-12-05T22:17:35.921 INFO:teuthology.task.install:Using flavor: default 2023-12-05T22:17:35.928 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-05T22:17:35.928 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-05T22:17:35.929 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-key list | grep Ceph 2023-12-05T22:17:35.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-key list | grep Ceph 2023-12-05T22:17:35.991 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-05T22:17:36.012 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-05T22:17:36.023 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-05T22:17:36.023 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph.com (release key) 2023-12-05T22:17:36.023 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-05T22:17:36.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4a50e96e7d9d85da28ef008617cb8342574bfd1b 2023-12-05T22:17:36.042 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-05T22:17:36.043 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph.com (release key) 2023-12-05T22:17:36.043 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-05T22:17:36.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4a50e96e7d9d85da28ef008617cb8342574bfd1b 2023-12-05T22:17:36.171 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default/ 2023-12-05T22:17:36.172 INFO:teuthology.task.install.deb:Package version is 19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:17:36.263 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default/ 2023-12-05T22:17:36.263 INFO:teuthology.task.install.deb:Package version is 19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:17:36.345 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:17:36.345 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-05T22:17:36.360 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2023-12-05T22:17:36.371 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:17:36.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-05T22:17:36.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2023-12-05T22:17:36.532 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-05T22:17:36.533 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-05T22:17:36.547 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-05T22:17:36.548 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-05T22:17:36.567 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-05T22:17:36.605 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy InRelease 2023-12-05T22:17:36.626 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-05T22:17:36.644 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy InRelease 2023-12-05T22:17:36.648 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-05T22:17:36.668 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-05T22:17:36.669 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2023-12-05T22:17:36.682 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-05T22:17:36.696 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-12-05T22:17:36.718 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-05T22:17:36.749 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-05T22:17:36.754 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2023-12-05T22:17:36.796 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-12-05T22:17:36.831 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-05T22:17:36.888 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 28.5 kB in 0s (82.1 kB/s) 2023-12-05T22:17:36.967 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 28.5 kB in 0s (69.2 kB/s) 2023-12-05T22:17:37.790 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-12-05T22:17:37.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-21-g4a50e96e-1jammy cephadm=19.0.0-21-g4a50e96e-1jammy ceph-mds=19.0.0-21-g4a50e96e-1jammy ceph-mgr=19.0.0-21-g4a50e96e-1jammy ceph-common=19.0.0-21-g4a50e96e-1jammy ceph-fuse=19.0.0-21-g4a50e96e-1jammy ceph-test=19.0.0-21-g4a50e96e-1jammy ceph-volume=19.0.0-21-g4a50e96e-1jammy radosgw=19.0.0-21-g4a50e96e-1jammy python3-rados=19.0.0-21-g4a50e96e-1jammy python3-rgw=19.0.0-21-g4a50e96e-1jammy python3-cephfs=19.0.0-21-g4a50e96e-1jammy python3-rbd=19.0.0-21-g4a50e96e-1jammy libcephfs2=19.0.0-21-g4a50e96e-1jammy libcephfs-dev=19.0.0-21-g4a50e96e-1jammy librados2=19.0.0-21-g4a50e96e-1jammy librbd1=19.0.0-21-g4a50e96e-1jammy rbd-fuse=19.0.0-21-g4a50e96e-1jammy rbd-mirror=19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:17:37.825 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-12-05T22:17:37.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-21-g4a50e96e-1jammy cephadm=19.0.0-21-g4a50e96e-1jammy ceph-mds=19.0.0-21-g4a50e96e-1jammy ceph-mgr=19.0.0-21-g4a50e96e-1jammy ceph-common=19.0.0-21-g4a50e96e-1jammy ceph-fuse=19.0.0-21-g4a50e96e-1jammy ceph-test=19.0.0-21-g4a50e96e-1jammy ceph-volume=19.0.0-21-g4a50e96e-1jammy radosgw=19.0.0-21-g4a50e96e-1jammy python3-rados=19.0.0-21-g4a50e96e-1jammy python3-rgw=19.0.0-21-g4a50e96e-1jammy python3-cephfs=19.0.0-21-g4a50e96e-1jammy python3-rbd=19.0.0-21-g4a50e96e-1jammy libcephfs2=19.0.0-21-g4a50e96e-1jammy libcephfs-dev=19.0.0-21-g4a50e96e-1jammy librados2=19.0.0-21-g4a50e96e-1jammy librbd1=19.0.0-21-g4a50e96e-1jammy rbd-fuse=19.0.0-21-g4a50e96e-1jammy rbd-mirror=19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:17:37.857 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-12-05T22:17:37.900 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-12-05T22:17:38.029 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2023-12-05T22:17:38.030 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2023-12-05T22:17:38.088 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2023-12-05T22:17:38.089 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2023-12-05T22:17:38.189 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2023-12-05T22:17:38.189 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-05T22:17:38.189 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-05T22:17:38.189 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-05T22:17:38.189 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-05T22:17:38.189 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-05T22:17:38.190 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-05T22:17:38.191 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2023-12-05T22:17:38.191 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-05T22:17:38.191 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-05T22:17:38.191 INFO:teuthology.orchestra.run.smithi062.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-05T22:17:38.191 INFO:teuthology.orchestra.run.smithi062.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-05T22:17:38.191 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-05T22:17:38.191 INFO:teuthology.orchestra.run.smithi062.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-05T22:17:38.192 INFO:teuthology.orchestra.run.smithi062.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet zip 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-05T22:17:38.193 INFO:teuthology.orchestra.run.smithi062.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-05T22:17:38.194 INFO:teuthology.orchestra.run.smithi062.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-05T22:17:38.194 INFO:teuthology.orchestra.run.smithi062.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-05T22:17:38.194 INFO:teuthology.orchestra.run.smithi062.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-05T22:17:38.194 INFO:teuthology.orchestra.run.smithi062.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-05T22:17:38.194 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2023-12-05T22:17:38.194 INFO:teuthology.orchestra.run.smithi062.stdout: btrfs-tools 2023-12-05T22:17:38.262 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2023-12-05T22:17:38.263 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-05T22:17:38.263 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-05T22:17:38.263 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-05T22:17:38.263 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-05T22:17:38.263 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-05T22:17:38.263 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-05T22:17:38.265 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2023-12-05T22:17:38.265 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-05T22:17:38.266 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet zip 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-05T22:17:38.267 INFO:teuthology.orchestra.run.smithi190.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-05T22:17:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-05T22:17:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-05T22:17:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-05T22:17:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-05T22:17:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-05T22:17:38.268 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2023-12-05T22:17:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: btrfs-tools 2023-12-05T22:17:38.296 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2023-12-05T22:17:38.296 INFO:teuthology.orchestra.run.smithi062.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-05T22:17:38.296 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-05T22:17:38.296 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-12-05T22:17:38.296 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2023-12-05T22:17:38.296 INFO:teuthology.orchestra.run.smithi062.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-12-05T22:17:38.296 INFO:teuthology.orchestra.run.smithi062.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2023-12-05T22:17:38.297 INFO:teuthology.orchestra.run.smithi062.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-05T22:17:38.298 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-12-05T22:17:38.299 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-12-05T22:17:38.300 INFO:teuthology.orchestra.run.smithi062.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-12-05T22:17:38.300 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-12-05T22:17:38.300 INFO:teuthology.orchestra.run.smithi062.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2023-12-05T22:17:38.367 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2023-12-05T22:17:38.367 INFO:teuthology.orchestra.run.smithi190.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-05T22:17:38.367 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-05T22:17:38.367 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-12-05T22:17:38.367 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2023-12-05T22:17:38.367 INFO:teuthology.orchestra.run.smithi190.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-12-05T22:17:38.368 INFO:teuthology.orchestra.run.smithi190.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2023-12-05T22:17:38.369 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2023-12-05T22:17:38.370 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-12-05T22:17:38.370 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-12-05T22:17:38.370 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-12-05T22:17:38.370 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-12-05T22:17:38.370 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-12-05T22:17:38.370 INFO:teuthology.orchestra.run.smithi190.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2023-12-05T22:17:38.425 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 119 newly installed, 0 to remove and 262 not upgraded. 2023-12-05T22:17:38.425 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 190 MB of archives. 2023-12-05T22:17:38.425 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 847 MB of additional disk space will be used. 2023-12-05T22:17:38.426 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-05T22:17:38.483 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 119 newly installed, 0 to remove and 262 not upgraded. 2023-12-05T22:17:38.484 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 190 MB of archives. 2023-12-05T22:17:38.484 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 847 MB of additional disk space will be used. 2023-12-05T22:17:38.484 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-05T22:17:38.509 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-21-g4a50e96e-1jammy [3,496 kB] 2023-12-05T22:17:38.512 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-05T22:17:38.515 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-05T22:17:38.532 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-05T22:17:38.539 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-21-g4a50e96e-1jammy [3,496 kB] 2023-12-05T22:17:38.590 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-05T22:17:38.595 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-05T22:17:38.608 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-05T22:17:38.612 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-05T22:17:38.613 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-05T22:17:38.613 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-05T22:17:38.622 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-05T22:17:38.624 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-05T22:17:38.626 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-05T22:17:38.627 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-05T22:17:38.628 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-05T22:17:38.631 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-05T22:17:38.645 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-05T22:17:38.647 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-05T22:17:38.648 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-05T22:17:38.648 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-05T22:17:38.649 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-05T22:17:38.649 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-05T22:17:38.650 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-05T22:17:38.650 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-05T22:17:38.663 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-05T22:17:38.663 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-05T22:17:38.680 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-05T22:17:38.682 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-05T22:17:38.683 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-05T22:17:38.683 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-05T22:17:38.684 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-05T22:17:38.685 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-05T22:17:38.685 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-05T22:17:38.685 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-05T22:17:38.698 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-05T22:17:38.698 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-05T22:17:38.714 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-05T22:17:38.720 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-05T22:17:38.720 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-05T22:17:38.720 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-05T22:17:38.721 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-05T22:17:38.722 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-05T22:17:38.724 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-05T22:17:38.725 INFO:teuthology.orchestra.run.smithi190.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-05T22:17:38.731 INFO:teuthology.orchestra.run.smithi190.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-05T22:17:38.732 INFO:teuthology.orchestra.run.smithi190.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-05T22:17:38.749 INFO:teuthology.orchestra.run.smithi190.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-05T22:17:38.749 INFO:teuthology.orchestra.run.smithi190.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-05T22:17:38.750 INFO:teuthology.orchestra.run.smithi190.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-05T22:17:38.752 INFO:teuthology.orchestra.run.smithi190.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-05T22:17:38.805 INFO:teuthology.orchestra.run.smithi190.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-05T22:17:38.806 INFO:teuthology.orchestra.run.smithi190.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-05T22:17:38.806 INFO:teuthology.orchestra.run.smithi190.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-05T22:17:38.807 INFO:teuthology.orchestra.run.smithi190.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-05T22:17:38.808 INFO:teuthology.orchestra.run.smithi190.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-05T22:17:38.825 INFO:teuthology.orchestra.run.smithi190.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-05T22:17:38.829 INFO:teuthology.orchestra.run.smithi190.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-05T22:17:38.829 INFO:teuthology.orchestra.run.smithi190.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-05T22:17:38.842 INFO:teuthology.orchestra.run.smithi190.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-05T22:17:38.842 INFO:teuthology.orchestra.run.smithi190.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-05T22:17:38.842 INFO:teuthology.orchestra.run.smithi190.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-05T22:17:38.843 INFO:teuthology.orchestra.run.smithi190.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-05T22:17:38.843 INFO:teuthology.orchestra.run.smithi190.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-05T22:17:38.843 INFO:teuthology.orchestra.run.smithi190.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-05T22:17:38.845 INFO:teuthology.orchestra.run.smithi190.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-21-g4a50e96e-1jammy [3,136 kB] 2023-12-05T22:17:38.857 INFO:teuthology.orchestra.run.smithi190.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-05T22:17:38.858 INFO:teuthology.orchestra.run.smithi190.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-05T22:17:38.860 INFO:teuthology.orchestra.run.smithi190.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-05T22:17:38.876 INFO:teuthology.orchestra.run.smithi190.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-05T22:17:38.877 INFO:teuthology.orchestra.run.smithi190.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-05T22:17:38.878 INFO:teuthology.orchestra.run.smithi190.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-05T22:17:38.880 INFO:teuthology.orchestra.run.smithi190.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-05T22:17:38.883 INFO:teuthology.orchestra.run.smithi190.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-05T22:17:38.884 INFO:teuthology.orchestra.run.smithi190.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-05T22:17:38.884 INFO:teuthology.orchestra.run.smithi190.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-05T22:17:38.893 INFO:teuthology.orchestra.run.smithi190.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-05T22:17:38.893 INFO:teuthology.orchestra.run.smithi190.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-05T22:17:38.896 INFO:teuthology.orchestra.run.smithi190.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-05T22:17:38.905 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-05T22:17:38.910 INFO:teuthology.orchestra.run.smithi190.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-05T22:17:38.911 INFO:teuthology.orchestra.run.smithi190.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-05T22:17:38.911 INFO:teuthology.orchestra.run.smithi190.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-05T22:17:38.913 INFO:teuthology.orchestra.run.smithi190.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-05T22:17:38.913 INFO:teuthology.orchestra.run.smithi190.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-05T22:17:38.918 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-05T22:17:38.919 INFO:teuthology.orchestra.run.smithi190.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-05T22:17:38.919 INFO:teuthology.orchestra.run.smithi190.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-05T22:17:38.929 INFO:teuthology.orchestra.run.smithi190.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-05T22:17:38.931 INFO:teuthology.orchestra.run.smithi190.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-05T22:17:38.931 INFO:teuthology.orchestra.run.smithi190.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-05T22:17:38.997 INFO:teuthology.orchestra.run.smithi190.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-21-g4a50e96e-1jammy [688 kB] 2023-12-05T22:17:38.999 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-05T22:17:39.022 INFO:teuthology.orchestra.run.smithi190.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-21-g4a50e96e-1jammy [356 kB] 2023-12-05T22:17:39.038 INFO:teuthology.orchestra.run.smithi190.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-21-g4a50e96e-1jammy [32.8 kB] 2023-12-05T22:17:39.038 INFO:teuthology.orchestra.run.smithi190.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-21-g4a50e96e-1jammy [180 kB] 2023-12-05T22:17:39.044 INFO:teuthology.orchestra.run.smithi190.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-21-g4a50e96e-1jammy [63.5 kB] 2023-12-05T22:17:39.046 INFO:teuthology.orchestra.run.smithi190.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-21-g4a50e96e-1jammy [330 kB] 2023-12-05T22:17:39.058 INFO:teuthology.orchestra.run.smithi190.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-21-g4a50e96e-1jammy [6,218 kB] 2023-12-05T22:17:39.231 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-05T22:17:39.241 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-05T22:17:39.271 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-05T22:17:39.283 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-05T22:17:39.284 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-05T22:17:39.285 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-05T22:17:39.293 INFO:teuthology.orchestra.run.smithi190.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-21-g4a50e96e-1jammy [112 kB] 2023-12-05T22:17:39.296 INFO:teuthology.orchestra.run.smithi190.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-21-g4a50e96e-1jammy [439 kB] 2023-12-05T22:17:39.304 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-05T22:17:39.312 INFO:teuthology.orchestra.run.smithi190.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-21-g4a50e96e-1jammy [22.6 MB] 2023-12-05T22:17:39.472 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-05T22:17:39.588 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-21-g4a50e96e-1jammy [3,136 kB] 2023-12-05T22:17:39.884 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-05T22:17:39.918 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-05T22:17:39.920 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-05T22:17:39.999 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-05T22:17:40.051 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-05T22:17:40.073 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-05T22:17:40.079 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-05T22:17:40.080 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-05T22:17:40.092 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-05T22:17:40.093 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-05T22:17:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-05T22:17:40.266 INFO:teuthology.orchestra.run.smithi190.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-21-g4a50e96e-1jammy [6,941 kB] 2023-12-05T22:17:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-21-g4a50e96e-1jammy [688 kB] 2023-12-05T22:17:40.342 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-05T22:17:40.371 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-05T22:17:40.388 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-05T22:17:40.404 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-21-g4a50e96e-1jammy [356 kB] 2023-12-05T22:17:40.432 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-05T22:17:40.448 INFO:teuthology.orchestra.run.smithi190.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-21-g4a50e96e-1jammy [238 kB] 2023-12-05T22:17:40.454 INFO:teuthology.orchestra.run.smithi190.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-21-g4a50e96e-1jammy [128 kB] 2023-12-05T22:17:40.463 INFO:teuthology.orchestra.run.smithi190.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-21-g4a50e96e-1jammy [1,496 kB] 2023-12-05T22:17:40.473 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-21-g4a50e96e-1jammy [32.8 kB] 2023-12-05T22:17:40.475 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-21-g4a50e96e-1jammy [180 kB] 2023-12-05T22:17:40.486 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-21-g4a50e96e-1jammy [63.5 kB] 2023-12-05T22:17:40.513 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-21-g4a50e96e-1jammy [330 kB] 2023-12-05T22:17:40.522 INFO:teuthology.orchestra.run.smithi190.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-21-g4a50e96e-1jammy [9,965 kB] 2023-12-05T22:17:40.558 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-21-g4a50e96e-1jammy [6,218 kB] 2023-12-05T22:17:40.643 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-05T22:17:40.658 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-05T22:17:40.665 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-05T22:17:40.671 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-05T22:17:40.692 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-05T22:17:40.695 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-05T22:17:40.861 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-05T22:17:40.946 INFO:teuthology.orchestra.run.smithi190.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-21-g4a50e96e-1jammy [31.9 MB] 2023-12-05T22:17:41.177 INFO:teuthology.orchestra.run.smithi062.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-05T22:17:41.214 INFO:teuthology.orchestra.run.smithi062.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-05T22:17:41.225 INFO:teuthology.orchestra.run.smithi062.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-05T22:17:41.407 INFO:teuthology.orchestra.run.smithi062.stdout:Get:47 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-21-g4a50e96e-1jammy [112 kB] 2023-12-05T22:17:41.421 INFO:teuthology.orchestra.run.smithi062.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-21-g4a50e96e-1jammy [439 kB] 2023-12-05T22:17:41.429 INFO:teuthology.orchestra.run.smithi062.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-05T22:17:41.429 INFO:teuthology.orchestra.run.smithi062.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-05T22:17:41.435 INFO:teuthology.orchestra.run.smithi062.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-05T22:17:41.441 INFO:teuthology.orchestra.run.smithi062.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-05T22:17:41.456 INFO:teuthology.orchestra.run.smithi062.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-05T22:17:41.470 INFO:teuthology.orchestra.run.smithi062.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-21-g4a50e96e-1jammy [22.6 MB] 2023-12-05T22:17:41.478 INFO:teuthology.orchestra.run.smithi062.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-05T22:17:41.483 INFO:teuthology.orchestra.run.smithi062.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-05T22:17:41.659 INFO:teuthology.orchestra.run.smithi062.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-05T22:17:41.926 INFO:teuthology.orchestra.run.smithi062.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-05T22:17:42.104 INFO:teuthology.orchestra.run.smithi062.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-05T22:17:42.119 INFO:teuthology.orchestra.run.smithi062.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-05T22:17:42.140 INFO:teuthology.orchestra.run.smithi062.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-05T22:17:42.204 INFO:teuthology.orchestra.run.smithi062.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-05T22:17:42.493 INFO:teuthology.orchestra.run.smithi190.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-21-g4a50e96e-1jammy [14.0 kB] 2023-12-05T22:17:42.494 INFO:teuthology.orchestra.run.smithi190.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-21-g4a50e96e-1jammy [840 kB] 2023-12-05T22:17:42.502 INFO:teuthology.orchestra.run.smithi062.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-05T22:17:42.504 INFO:teuthology.orchestra.run.smithi062.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-05T22:17:42.507 INFO:teuthology.orchestra.run.smithi062.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-05T22:17:42.508 INFO:teuthology.orchestra.run.smithi062.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-05T22:17:42.510 INFO:teuthology.orchestra.run.smithi190.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-21-g4a50e96e-1jammy [2,147 kB] 2023-12-05T22:17:42.510 INFO:teuthology.orchestra.run.smithi062.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-05T22:17:42.548 INFO:teuthology.orchestra.run.smithi190.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-21-g4a50e96e-1jammy [826 kB] 2023-12-05T22:17:42.576 INFO:teuthology.orchestra.run.smithi190.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-21-g4a50e96e-1jammy [139 kB] 2023-12-05T22:17:42.580 INFO:teuthology.orchestra.run.smithi190.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-21-g4a50e96e-1jammy [1,904 kB] 2023-12-05T22:17:42.673 INFO:teuthology.orchestra.run.smithi190.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-21-g4a50e96e-1jammy [8,625 kB] 2023-12-05T22:17:42.719 INFO:teuthology.orchestra.run.smithi062.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-05T22:17:43.076 INFO:teuthology.orchestra.run.smithi190.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-21-g4a50e96e-1jammy [14.2 kB] 2023-12-05T22:17:43.076 INFO:teuthology.orchestra.run.smithi190.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-21-g4a50e96e-1jammy [48.8 MB] 2023-12-05T22:17:43.150 INFO:teuthology.orchestra.run.smithi062.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-05T22:17:43.156 INFO:teuthology.orchestra.run.smithi062.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-05T22:17:43.445 INFO:teuthology.orchestra.run.smithi062.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-05T22:17:43.446 INFO:teuthology.orchestra.run.smithi062.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-05T22:17:43.448 INFO:teuthology.orchestra.run.smithi062.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-05T22:17:43.450 INFO:teuthology.orchestra.run.smithi062.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-05T22:17:43.452 INFO:teuthology.orchestra.run.smithi062.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-05T22:17:43.453 INFO:teuthology.orchestra.run.smithi062.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-05T22:17:43.455 INFO:teuthology.orchestra.run.smithi062.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-05T22:17:43.456 INFO:teuthology.orchestra.run.smithi062.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-05T22:17:43.637 INFO:teuthology.orchestra.run.smithi062.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-05T22:17:43.964 INFO:teuthology.orchestra.run.smithi062.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-21-g4a50e96e-1jammy [6,941 kB] 2023-12-05T22:17:44.044 INFO:teuthology.orchestra.run.smithi062.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-05T22:17:44.091 INFO:teuthology.orchestra.run.smithi062.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-05T22:17:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-05T22:17:44.223 INFO:teuthology.orchestra.run.smithi062.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-05T22:17:44.262 INFO:teuthology.orchestra.run.smithi062.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-05T22:17:44.274 INFO:teuthology.orchestra.run.smithi062.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-05T22:17:44.279 INFO:teuthology.orchestra.run.smithi062.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-05T22:17:44.327 INFO:teuthology.orchestra.run.smithi062.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-05T22:17:44.330 INFO:teuthology.orchestra.run.smithi062.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-05T22:17:44.351 INFO:teuthology.orchestra.run.smithi062.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-05T22:17:44.506 INFO:teuthology.orchestra.run.smithi062.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-21-g4a50e96e-1jammy [238 kB] 2023-12-05T22:17:44.517 INFO:teuthology.orchestra.run.smithi062.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-05T22:17:44.526 INFO:teuthology.orchestra.run.smithi062.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-21-g4a50e96e-1jammy [128 kB] 2023-12-05T22:17:44.537 INFO:teuthology.orchestra.run.smithi062.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-21-g4a50e96e-1jammy [1,496 kB] 2023-12-05T22:17:44.668 INFO:teuthology.orchestra.run.smithi062.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-21-g4a50e96e-1jammy [9,965 kB] 2023-12-05T22:17:44.782 INFO:teuthology.orchestra.run.smithi062.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-05T22:17:44.800 INFO:teuthology.orchestra.run.smithi062.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-05T22:17:44.953 INFO:teuthology.orchestra.run.smithi062.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-05T22:17:44.982 INFO:teuthology.orchestra.run.smithi062.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-05T22:17:45.140 INFO:teuthology.orchestra.run.smithi062.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-05T22:17:45.143 INFO:teuthology.orchestra.run.smithi062.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-05T22:17:45.144 INFO:teuthology.orchestra.run.smithi062.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-05T22:17:45.167 INFO:teuthology.orchestra.run.smithi062.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-05T22:17:45.172 INFO:teuthology.orchestra.run.smithi062.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-05T22:17:45.532 INFO:teuthology.orchestra.run.smithi062.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-21-g4a50e96e-1jammy [31.9 MB] 2023-12-05T22:17:45.828 INFO:teuthology.orchestra.run.smithi190.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-21-g4a50e96e-1jammy [120 kB] 2023-12-05T22:17:45.830 INFO:teuthology.orchestra.run.smithi190.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-21-g4a50e96e-1jammy [38.3 kB] 2023-12-05T22:17:45.831 INFO:teuthology.orchestra.run.smithi190.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-21-g4a50e96e-1jammy [12.3 MB] 2023-12-05T22:17:46.398 INFO:teuthology.orchestra.run.smithi190.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-21-g4a50e96e-1jammy [92.0 kB] 2023-12-05T22:17:46.442 INFO:teuthology.orchestra.run.smithi190.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-21-g4a50e96e-1jammy [3,073 kB] 2023-12-05T22:17:47.084 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 190 MB in 8s (23.3 MB/s) 2023-12-05T22:17:47.238 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-05T22:17:47.328 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165983 files and directories currently installed.) 2023-12-05T22:17:47.333 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-05T22:17:47.428 INFO:teuthology.orchestra.run.smithi062.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-21-g4a50e96e-1jammy [14.0 kB] 2023-12-05T22:17:47.428 INFO:teuthology.orchestra.run.smithi062.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-21-g4a50e96e-1jammy [840 kB] 2023-12-05T22:17:47.434 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-05T22:17:47.445 INFO:teuthology.orchestra.run.smithi062.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-21-g4a50e96e-1jammy [2,147 kB] 2023-12-05T22:17:47.484 INFO:teuthology.orchestra.run.smithi062.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-21-g4a50e96e-1jammy [826 kB] 2023-12-05T22:17:47.508 INFO:teuthology.orchestra.run.smithi062.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-21-g4a50e96e-1jammy [139 kB] 2023-12-05T22:17:47.513 INFO:teuthology.orchestra.run.smithi062.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-21-g4a50e96e-1jammy [1,904 kB] 2023-12-05T22:17:47.608 INFO:teuthology.orchestra.run.smithi062.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-21-g4a50e96e-1jammy [8,625 kB] 2023-12-05T22:17:47.780 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-05T22:17:47.806 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-05T22:17:47.846 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-05T22:17:47.969 INFO:teuthology.orchestra.run.smithi062.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-21-g4a50e96e-1jammy [14.2 kB] 2023-12-05T22:17:47.969 INFO:teuthology.orchestra.run.smithi062.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-21-g4a50e96e-1jammy [48.8 MB] 2023-12-05T22:17:48.182 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-05T22:17:48.208 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-05T22:17:48.249 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-05T22:17:48.594 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-05T22:17:48.620 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-05T22:17:48.660 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:17:49.047 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-05T22:17:49.074 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-05T22:17:49.113 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:17:49.458 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-05T22:17:49.485 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-05T22:17:49.524 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:17:49.828 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-05T22:17:49.854 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-05T22:17:49.886 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-05T22:17:50.197 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librados2. 2023-12-05T22:17:50.224 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../007-librados2_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:50.263 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librados2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:50.733 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librbd1. 2023-12-05T22:17:50.749 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../008-librbd1_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:50.782 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librbd1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:51.112 INFO:teuthology.orchestra.run.smithi062.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-21-g4a50e96e-1jammy [120 kB] 2023-12-05T22:17:51.115 INFO:teuthology.orchestra.run.smithi062.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-21-g4a50e96e-1jammy [38.3 kB] 2023-12-05T22:17:51.116 INFO:teuthology.orchestra.run.smithi062.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-21-g4a50e96e-1jammy [12.3 MB] 2023-12-05T22:17:51.244 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs2. 2023-12-05T22:17:51.270 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:51.310 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:51.613 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rados. 2023-12-05T22:17:51.628 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../010-python3-rados_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:51.671 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rados (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:51.935 INFO:teuthology.orchestra.run.smithi062.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-21-g4a50e96e-1jammy [92.0 kB] 2023-12-05T22:17:51.938 INFO:teuthology.orchestra.run.smithi062.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing/4a50e96e7d9d85da28ef008617cb8342574bfd1b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-21-g4a50e96e-1jammy [3,073 kB] 2023-12-05T22:17:52.082 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-05T22:17:52.108 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:17:52.148 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-argparse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:52.393 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cephfs. 2023-12-05T22:17:52.408 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:52.443 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cephfs (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:52.731 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 190 MB in 14s (13.7 MB/s) 2023-12-05T22:17:52.754 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-05T22:17:52.780 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:17:52.820 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:52.895 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-05T22:17:52.997 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165983 files and directories currently installed.) 2023-12-05T22:17:53.002 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-05T22:17:53.094 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-05T22:17:53.148 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-05T22:17:53.174 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-05T22:17:53.215 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-05T22:17:53.447 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-05T22:17:53.474 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-05T22:17:53.514 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-05T22:17:53.526 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-prettytable. 2023-12-05T22:17:53.552 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-12-05T22:17:53.593 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-05T22:17:53.850 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-05T22:17:53.853 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rbd. 2023-12-05T22:17:53.868 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:53.877 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-05T22:17:53.903 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rbd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:53.916 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-05T22:17:54.198 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-05T22:17:54.224 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-05T22:17:54.264 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-05T22:17:54.287 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-05T22:17:54.314 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-05T22:17:54.353 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:17:54.558 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-05T22:17:54.574 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-05T22:17:54.617 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-05T22:17:54.757 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-05T22:17:54.783 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-05T22:17:54.842 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:17:54.928 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-05T22:17:54.947 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-05T22:17:54.986 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-05T22:17:55.202 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-05T22:17:55.228 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-05T22:17:55.268 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:17:55.331 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua5.1. 2023-12-05T22:17:55.357 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-05T22:17:55.397 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-05T22:17:55.580 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-05T22:17:55.595 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-05T22:17:55.630 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-05T22:17:55.725 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-any. 2023-12-05T22:17:55.741 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2023-12-05T22:17:55.784 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-05T22:17:55.958 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librados2. 2023-12-05T22:17:55.984 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../007-librados2_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:56.024 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librados2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:56.036 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package zip. 2023-12-05T22:17:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2023-12-05T22:17:56.094 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking zip (3.0-12build2) ... 2023-12-05T22:17:56.422 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package unzip. 2023-12-05T22:17:56.449 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-05T22:17:56.477 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librbd1. 2023-12-05T22:17:56.489 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-05T22:17:56.504 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../008-librbd1_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:56.544 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librbd1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:56.963 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs2. 2023-12-05T22:17:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:57.013 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:57.118 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package luarocks. 2023-12-05T22:17:57.133 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-05T22:17:57.167 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-05T22:17:57.308 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rados. 2023-12-05T22:17:57.334 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../010-python3-rados_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:57.374 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rados (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:57.492 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librgw2. 2023-12-05T22:17:57.518 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../025-librgw2_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:57.567 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librgw2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:57.702 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-05T22:17:57.729 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:17:57.769 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-argparse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:58.021 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cephfs. 2023-12-05T22:17:58.037 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:58.079 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cephfs (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:58.086 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rgw. 2023-12-05T22:17:58.101 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:58.144 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:58.332 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-05T22:17:58.348 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:17:58.382 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:58.480 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-05T22:17:58.507 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-05T22:17:58.547 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-05T22:17:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-05T22:17:58.695 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-05T22:17:58.735 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-05T22:17:58.858 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libradosstriper1. 2023-12-05T22:17:58.884 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:58.918 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libradosstriper1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:59.063 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-prettytable. 2023-12-05T22:17:59.090 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-12-05T22:17:59.130 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-05T22:17:59.336 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-common. 2023-12-05T22:17:59.363 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../029-ceph-common_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:59.391 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rbd. 2023-12-05T22:17:59.402 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:59.406 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:17:59.449 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rbd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:17:59.794 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-05T22:17:59.820 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-05T22:17:59.860 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-05T22:18:00.163 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-05T22:18:00.178 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-05T22:18:00.221 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-05T22:18:00.499 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-05T22:18:00.526 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-05T22:18:00.558 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-05T22:18:00.674 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-base. 2023-12-05T22:18:00.701 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../030-ceph-base_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:00.750 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-base (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:00.894 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua5.1. 2023-12-05T22:18:00.921 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-05T22:18:00.960 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-05T22:18:01.297 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-any. 2023-12-05T22:18:01.324 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2023-12-05T22:18:01.364 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-05T22:18:01.445 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-05T22:18:01.471 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-05T22:18:01.511 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-05T22:18:01.592 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package zip. 2023-12-05T22:18:01.608 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2023-12-05T22:18:01.641 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking zip (3.0-12build2) ... 2023-12-05T22:18:01.773 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cheroot. 2023-12-05T22:18:01.799 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-05T22:18:01.831 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-05T22:18:01.936 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package unzip. 2023-12-05T22:18:01.963 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-05T22:18:02.121 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-05T22:18:02.150 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-05T22:18:02.177 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-05T22:18:02.217 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-05T22:18:02.520 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-05T22:18:02.546 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-05T22:18:02.586 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-05T22:18:02.665 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package luarocks. 2023-12-05T22:18:02.692 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-05T22:18:02.732 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-05T22:18:02.872 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-05T22:18:02.899 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-05T22:18:02.939 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-05T22:18:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librgw2. 2023-12-05T22:18:03.141 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../025-librgw2_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:03.189 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librgw2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:03.208 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempora. 2023-12-05T22:18:03.223 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2023-12-05T22:18:03.258 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-05T22:18:03.536 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-portend. 2023-12-05T22:18:03.562 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2023-12-05T22:18:03.602 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-05T22:18:03.667 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rgw. 2023-12-05T22:18:03.683 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:03.717 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:03.931 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-05T22:18:03.957 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-05T22:18:03.998 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-05T22:18:04.054 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-05T22:18:04.080 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-05T22:18:04.120 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-05T22:18:04.300 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-05T22:18:04.326 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-05T22:18:04.366 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-05T22:18:04.422 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libradosstriper1. 2023-12-05T22:18:04.449 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:04.498 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libradosstriper1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:04.712 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-natsort. 2023-12-05T22:18:04.738 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2023-12-05T22:18:04.779 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-05T22:18:04.859 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-common. 2023-12-05T22:18:04.886 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../029-ceph-common_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:04.917 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:05.089 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-logutils. 2023-12-05T22:18:05.105 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2023-12-05T22:18:05.147 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-05T22:18:05.442 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-05T22:18:05.468 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-05T22:18:05.508 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-05T22:18:05.778 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-mako. 2023-12-05T22:18:05.804 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-05T22:18:05.844 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-05T22:18:06.164 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-05T22:18:06.191 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-05T22:18:06.223 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-base. 2023-12-05T22:18:06.230 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-05T22:18:06.250 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../030-ceph-base_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:06.307 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-base (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:06.533 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-05T22:18:06.560 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-05T22:18:06.600 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-05T22:18:06.894 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webob. 2023-12-05T22:18:06.910 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-05T22:18:06.944 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-05T22:18:07.053 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-05T22:18:07.079 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-05T22:18:07.119 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-05T22:18:07.422 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cheroot. 2023-12-05T22:18:07.449 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-05T22:18:07.488 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-05T22:18:07.522 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-waitress. 2023-12-05T22:18:07.549 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-05T22:18:07.592 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-05T22:18:07.808 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-05T22:18:07.824 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-05T22:18:07.858 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-05T22:18:07.891 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempita. 2023-12-05T22:18:07.907 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-05T22:18:07.941 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-05T22:18:08.128 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-05T22:18:08.154 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-05T22:18:08.194 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-05T22:18:08.244 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-paste. 2023-12-05T22:18:08.271 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-05T22:18:08.311 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-05T22:18:08.497 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-05T22:18:08.524 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-05T22:18:08.564 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-05T22:18:08.589 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-05T22:18:08.604 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-05T22:18:08.647 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-05T22:18:08.827 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempora. 2023-12-05T22:18:08.854 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2023-12-05T22:18:08.885 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-05T22:18:08.933 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-05T22:18:08.959 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-05T22:18:08.999 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-05T22:18:09.188 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-portend. 2023-12-05T22:18:09.215 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2023-12-05T22:18:09.254 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-05T22:18:09.294 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webtest. 2023-12-05T22:18:09.321 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2023-12-05T22:18:09.360 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-05T22:18:09.557 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-05T22:18:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-05T22:18:09.623 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-05T22:18:09.672 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pecan. 2023-12-05T22:18:09.698 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-05T22:18:09.738 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-05T22:18:09.910 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-05T22:18:09.937 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-05T22:18:09.976 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-05T22:18:10.118 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-certifi. 2023-12-05T22:18:10.145 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2023-12-05T22:18:10.184 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-05T22:18:10.330 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-natsort. 2023-12-05T22:18:10.357 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2023-12-05T22:18:10.396 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-05T22:18:10.454 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-idna. 2023-12-05T22:18:10.469 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2023-12-05T22:18:10.504 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-05T22:18:10.707 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-logutils. 2023-12-05T22:18:10.734 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2023-12-05T22:18:10.774 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-05T22:18:10.798 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-urllib3. 2023-12-05T22:18:10.825 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-05T22:18:10.865 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-05T22:18:11.060 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-05T22:18:11.076 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-05T22:18:11.110 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-05T22:18:11.168 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests. 2023-12-05T22:18:11.195 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-05T22:18:11.235 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-05T22:18:11.379 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-mako. 2023-12-05T22:18:11.406 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-05T22:18:11.446 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-05T22:18:11.546 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-05T22:18:11.572 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-05T22:18:11.612 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-05T22:18:12.366 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-05T22:18:12.393 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-05T22:18:12.466 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-05T22:18:12.774 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-05T22:18:12.800 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:12.840 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-modules-core (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:13.010 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-05T22:18:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-05T22:18:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-05T22:18:13.228 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-05T22:18:13.255 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:13.295 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:13.338 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webob. 2023-12-05T22:18:13.354 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-05T22:18:13.388 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-05T22:18:13.539 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-05T22:18:13.554 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-05T22:18:13.589 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-05T22:18:13.716 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-waitress. 2023-12-05T22:18:13.743 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-05T22:18:13.785 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-05T22:18:13.850 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-openssl. 2023-12-05T22:18:13.876 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2023-12-05T22:18:13.916 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-05T22:18:14.052 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempita. 2023-12-05T22:18:14.068 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-05T22:18:14.110 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-05T22:18:14.270 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr. 2023-12-05T22:18:14.296 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:14.336 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:14.413 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-paste. 2023-12-05T22:18:14.440 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-05T22:18:14.479 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-05T22:18:14.733 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mon. 2023-12-05T22:18:14.757 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:14.797 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mon (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:14.807 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-05T22:18:14.823 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-05T22:18:14.865 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-05T22:18:15.152 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-05T22:18:15.179 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-05T22:18:15.218 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-05T22:18:15.488 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webtest. 2023-12-05T22:18:15.515 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2023-12-05T22:18:15.546 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-05T22:18:15.617 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-osd. 2023-12-05T22:18:15.644 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:15.683 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-osd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:15.840 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pecan. 2023-12-05T22:18:15.867 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-05T22:18:15.907 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-05T22:18:16.270 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-certifi. 2023-12-05T22:18:16.297 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2023-12-05T22:18:16.336 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-05T22:18:16.656 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-idna. 2023-12-05T22:18:16.683 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2023-12-05T22:18:16.722 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-05T22:18:17.033 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-urllib3. 2023-12-05T22:18:17.062 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-05T22:18:17.100 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-05T22:18:17.420 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests. 2023-12-05T22:18:17.447 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-05T22:18:17.486 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-05T22:18:17.697 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph. 2023-12-05T22:18:17.724 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../066-ceph_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:17.781 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-05T22:18:17.809 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-05T22:18:17.847 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-05T22:18:17.853 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:18.150 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-05T22:18:18.172 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-fuse. 2023-12-05T22:18:18.177 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:18.199 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:18.217 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-modules-core (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:18.239 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:18.567 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mds. 2023-12-05T22:18:18.580 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-05T22:18:18.594 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:18.596 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:18.625 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mds (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:18.629 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:18.949 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-05T22:18:18.976 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-05T22:18:18.995 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package cephadm. 2023-12-05T22:18:19.015 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-05T22:18:19.022 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../069-cephadm_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:19.061 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:19.319 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-openssl. 2023-12-05T22:18:19.346 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2023-12-05T22:18:19.381 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-05T22:18:19.385 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-05T22:18:19.407 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-05T22:18:19.447 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-05T22:18:19.729 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr. 2023-12-05T22:18:19.757 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:19.775 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-05T22:18:19.796 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:19.801 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-05T22:18:19.841 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-05T22:18:20.141 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mon. 2023-12-05T22:18:20.168 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:20.199 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mon (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:21.005 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-babel. 2023-12-05T22:18:21.032 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-05T22:18:21.071 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-05T22:18:21.161 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-osd. 2023-12-05T22:18:21.188 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:21.227 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-osd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:21.416 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jinja2. 2023-12-05T22:18:21.442 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1_all.deb ... 2023-12-05T22:18:21.482 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-05T22:18:21.844 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-05T22:18:21.870 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:21.910 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:22.255 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-05T22:18:22.281 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2023-12-05T22:18:22.321 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-05T22:18:22.598 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-routes. 2023-12-05T22:18:22.614 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-05T22:18:22.648 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-05T22:18:23.027 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-05T22:18:23.054 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:23.093 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-dashboard (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:23.263 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph. 2023-12-05T22:18:23.280 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../066-ceph_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:23.322 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:23.547 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-05T22:18:23.563 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-05T22:18:23.599 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-fuse. 2023-12-05T22:18:23.604 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-05T22:18:23.615 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:23.649 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:24.011 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mds. 2023-12-05T22:18:24.038 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:24.077 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mds (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:24.125 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-joblib. 2023-12-05T22:18:24.153 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-05T22:18:24.191 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-05T22:18:24.472 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephadm. 2023-12-05T22:18:24.499 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../069-cephadm_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:24.538 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:24.538 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-05T22:18:24.566 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-05T22:18:24.596 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-05T22:18:24.857 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-05T22:18:24.884 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-05T22:18:24.899 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn. 2023-12-05T22:18:24.924 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-05T22:18:24.927 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-05T22:18:24.966 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-05T22:18:25.252 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-05T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-05T22:18:25.318 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-05T22:18:25.875 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-05T22:18:25.903 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:26.027 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:26.340 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-babel. 2023-12-05T22:18:26.367 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-05T22:18:26.406 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-05T22:18:26.708 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jinja2. 2023-12-05T22:18:26.733 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1_all.deb ... 2023-12-05T22:18:26.767 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-05T22:18:26.922 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cachetools. 2023-12-05T22:18:26.949 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2023-12-05T22:18:26.980 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-05T22:18:27.112 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-05T22:18:27.139 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:27.178 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:27.258 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-05T22:18:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-05T22:18:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-05T22:18:27.515 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-05T22:18:27.542 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2023-12-05T22:18:27.581 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-05T22:18:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-05T22:18:27.663 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-05T22:18:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-05T22:18:27.850 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-routes. 2023-12-05T22:18:27.867 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-05T22:18:27.900 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-05T22:18:27.997 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rsa. 2023-12-05T22:18:28.024 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2023-12-05T22:18:28.056 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-05T22:18:28.262 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-05T22:18:28.290 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:28.328 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-dashboard (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:28.576 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-google-auth. 2023-12-05T22:18:28.603 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2023-12-05T22:18:28.642 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-05T22:18:28.731 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-05T22:18:28.760 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-05T22:18:28.798 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-05T22:18:28.979 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-05T22:18:29.006 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-05T22:18:29.045 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-05T22:18:29.285 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-joblib. 2023-12-05T22:18:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-05T22:18:29.348 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-websocket. 2023-12-05T22:18:29.359 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-05T22:18:29.375 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2023-12-05T22:18:29.414 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-05T22:18:29.730 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-05T22:18:29.758 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-05T22:18:29.793 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-05T22:18:29.804 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-05T22:18:29.819 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-05T22:18:29.859 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-05T22:18:30.107 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn. 2023-12-05T22:18:30.135 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-05T22:18:30.173 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-05T22:18:30.656 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-05T22:18:30.684 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:30.723 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:30.848 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-05T22:18:30.865 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:30.975 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:31.017 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-05T22:18:31.045 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-05T22:18:31.083 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-05T22:18:31.403 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-05T22:18:31.431 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-05T22:18:31.470 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-05T22:18:31.747 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package jq. 2023-12-05T22:18:31.775 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-05T22:18:31.814 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-05T22:18:31.929 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cachetools. 2023-12-05T22:18:31.957 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2023-12-05T22:18:31.995 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-05T22:18:32.066 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package socat. 2023-12-05T22:18:32.094 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-05T22:18:32.133 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-05T22:18:32.298 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-05T22:18:32.326 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-05T22:18:32.364 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-05T22:18:32.444 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package xmlstarlet. 2023-12-05T22:18:32.460 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-05T22:18:32.502 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-05T22:18:32.709 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-05T22:18:32.737 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-05T22:18:32.776 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-05T22:18:32.865 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-test. 2023-12-05T22:18:32.881 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../097-ceph-test_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:32.923 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-test (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:33.104 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rsa. 2023-12-05T22:18:33.132 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2023-12-05T22:18:33.170 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-05T22:18:33.882 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-google-auth. 2023-12-05T22:18:33.910 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2023-12-05T22:18:33.940 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-05T22:18:34.244 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-05T22:18:34.272 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-05T22:18:34.310 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-05T22:18:34.621 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-websocket. 2023-12-05T22:18:34.649 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2023-12-05T22:18:34.688 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-05T22:18:34.999 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-05T22:18:35.027 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-05T22:18:35.065 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-05T22:18:35.185 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-volume. 2023-12-05T22:18:35.213 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:35.252 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-volume (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:35.581 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-05T22:18:35.608 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:35.647 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs-dev (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:35.763 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-05T22:18:35.779 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:35.837 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:35.933 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-05T22:18:35.949 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-05T22:18:35.983 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-05T22:18:36.082 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-05T22:18:36.110 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-05T22:18:36.140 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-05T22:18:36.313 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-05T22:18:36.340 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2023-12-05T22:18:36.379 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-05T22:18:36.476 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-05T22:18:36.504 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-05T22:18:36.542 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-05T22:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package nvme-cli. 2023-12-05T22:18:36.786 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-05T22:18:36.812 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package jq. 2023-12-05T22:18:36.824 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-05T22:18:36.828 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-05T22:18:36.870 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-05T22:18:37.097 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package socat. 2023-12-05T22:18:37.114 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-05T22:18:37.156 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-05T22:18:37.263 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package pkg-config. 2023-12-05T22:18:37.291 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-05T22:18:37.329 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-05T22:18:37.450 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package xmlstarlet. 2023-12-05T22:18:37.467 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-05T22:18:37.500 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-05T22:18:37.565 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-05T22:18:37.581 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-05T22:18:37.615 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-05T22:18:37.838 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-test. 2023-12-05T22:18:37.855 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../097-ceph-test_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:37.896 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-test (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:38.010 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-05T22:18:38.038 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-05T22:18:38.068 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-05T22:18:38.528 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastescript. 2023-12-05T22:18:38.555 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-05T22:18:38.654 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-05T22:18:38.974 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pluggy. 2023-12-05T22:18:39.002 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-05T22:18:39.040 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-05T22:18:39.351 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-psutil. 2023-12-05T22:18:39.379 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-05T22:18:39.418 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-05T22:18:39.746 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-py. 2023-12-05T22:18:39.774 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2023-12-05T22:18:39.812 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-05T22:18:40.116 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pygments. 2023-12-05T22:18:40.144 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-05T22:18:40.182 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-05T22:18:40.225 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-volume. 2023-12-05T22:18:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-21-g4a50e96e-1jammy_all.deb ... 2023-12-05T22:18:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-volume (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:40.595 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-05T22:18:40.624 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:40.662 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs-dev (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:40.670 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-05T22:18:40.697 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-05T22:18:40.735 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-05T22:18:40.998 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-05T22:18:41.026 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-05T22:18:41.055 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-toml. 2023-12-05T22:18:41.064 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-05T22:18:41.082 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2023-12-05T22:18:41.121 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-05T22:18:41.399 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pytest. 2023-12-05T22:18:41.402 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-05T22:18:41.427 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-05T22:18:41.430 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2023-12-05T22:18:41.457 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-05T22:18:41.477 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-05T22:18:41.802 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplejson. 2023-12-05T22:18:41.822 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package nvme-cli. 2023-12-05T22:18:41.830 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-05T22:18:41.851 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-05T22:18:41.869 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-05T22:18:41.889 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-05T22:18:42.155 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-05T22:18:42.171 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-05T22:18:42.213 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-05T22:18:42.277 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package pkg-config. 2023-12-05T22:18:42.302 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-05T22:18:42.344 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-05T22:18:42.571 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-05T22:18:42.589 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-05T22:18:42.629 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-05T22:18:42.917 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package radosgw. 2023-12-05T22:18:42.945 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../116-radosgw_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:42.983 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking radosgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:42.999 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-05T22:18:43.028 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-05T22:18:43.057 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-05T22:18:43.360 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastescript. 2023-12-05T22:18:43.389 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-05T22:18:43.426 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-05T22:18:43.761 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package rbd-fuse. 2023-12-05T22:18:43.778 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:43.780 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pluggy. 2023-12-05T22:18:43.808 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-05T22:18:43.819 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking rbd-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:43.846 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-05T22:18:44.164 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package rbd-mirror. 2023-12-05T22:18:44.192 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:44.230 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking rbd-mirror (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:44.366 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-psutil. 2023-12-05T22:18:44.394 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-05T22:18:44.432 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-05T22:18:44.701 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-05T22:18:44.777 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-py. 2023-12-05T22:18:44.805 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2023-12-05T22:18:44.854 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-05T22:18:44.952 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-05T22:18:45.078 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-05T22:18:45.172 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pygments. 2023-12-05T22:18:45.200 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-05T22:18:45.238 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-05T22:18:45.352 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-05T22:18:45.650 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-05T22:18:45.679 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-05T22:18:45.707 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-05T22:18:45.708 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-05T22:18:46.020 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-toml. 2023-12-05T22:18:46.048 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2023-12-05T22:18:46.094 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-05T22:18:46.121 INFO:teuthology.orchestra.run.smithi190.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-05T22:18:46.171 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:46.397 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pytest. 2023-12-05T22:18:46.425 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-05T22:18:46.463 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-05T22:18:46.623 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user cephadm....done 2023-12-05T22:18:46.777 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-05T22:18:46.800 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplejson. 2023-12-05T22:18:46.829 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-05T22:18:46.866 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-05T22:18:47.054 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-05T22:18:47.136 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-05T22:18:47.153 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-05T22:18:47.202 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-05T22:18:47.305 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-05T22:18:47.430 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-05T22:18:47.665 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-05T22:18:47.908 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-05T22:18:47.931 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package radosgw. 2023-12-05T22:18:47.960 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../116-radosgw_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:47.997 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking radosgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:48.034 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-05T22:18:48.335 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-05T22:18:48.696 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-05T22:18:48.750 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package rbd-fuse. 2023-12-05T22:18:48.767 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:48.800 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking rbd-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:48.955 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-05T22:18:49.138 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package rbd-mirror. 2023-12-05T22:18:49.167 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-21-g4a50e96e-1jammy_amd64.deb ... 2023-12-05T22:18:49.205 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking rbd-mirror (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:49.231 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-05T22:18:49.508 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-05T22:18:49.651 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-05T22:18:49.758 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-05T22:18:49.910 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-05T22:18:50.010 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-05T22:18:50.018 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-05T22:18:50.254 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-05T22:18:50.269 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-argparse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:50.537 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-05T22:18:50.654 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-05T22:18:50.821 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-05T22:18:51.057 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-05T22:18:51.057 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-05T22:18:51.144 INFO:teuthology.orchestra.run.smithi062.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-05T22:18:51.246 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:51.291 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-05T22:18:51.675 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-05T22:18:51.705 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-05T22:18:51.738 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user cephadm....done 2023-12-05T22:18:51.769 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-05T22:18:51.893 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-05T22:18:51.894 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-05T22:18:52.169 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-05T22:18:52.263 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-05T22:18:52.389 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-05T22:18:52.487 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-05T22:18:52.613 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-05T22:18:52.650 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-05T22:18:52.864 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-05T22:18:52.916 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-05T22:18:53.124 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-05T22:18:53.217 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up zip (3.0-12build2) ... 2023-12-05T22:18:53.250 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-05T22:18:53.343 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-05T22:18:53.559 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-05T22:18:53.928 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-05T22:18:54.188 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-05T22:18:54.246 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-05T22:18:54.472 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-05T22:18:54.521 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-05T22:18:54.639 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-05T22:18:54.690 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-05T22:18:54.748 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-05T22:18:54.933 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-05T22:18:55.032 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-05T22:18:55.217 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-05T22:18:55.417 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-05T22:18:55.528 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-argparse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:55.668 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-05T22:18:55.869 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-05T22:18:55.987 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-05T22:18:56.062 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-05T22:18:56.122 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-05T22:18:56.247 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-05T22:18:56.354 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-05T22:18:56.540 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-05T22:18:56.689 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-05T22:18:56.881 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-05T22:18:56.913 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-05T22:18:56.932 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-05T22:18:56.985 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-05T22:18:57.058 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:18:57.110 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-05T22:18:57.359 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-idna (3.3-1) ... 2023-12-05T22:18:57.486 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-05T22:18:57.621 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-05T22:18:57.677 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-05T22:18:57.897 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-05T22:18:58.148 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-05T22:18:58.449 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up zip (3.0-12build2) ... 2023-12-05T22:18:58.575 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-05T22:18:58.874 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-05T22:18:59.125 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:18:59.260 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-05T22:18:59.310 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-05T22:18:59.519 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-05T22:18:59.587 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-05T22:18:59.662 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-05T22:18:59.788 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-05T22:19:00.012 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-05T22:19:00.072 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-05T22:19:00.139 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-05T22:19:00.315 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-05T22:19:00.431 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-05T22:19:00.550 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:19:00.675 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-05T22:19:00.841 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-05T22:19:00.977 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-05T22:19:01.083 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-05T22:19:01.236 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-05T22:19:01.493 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-05T22:19:01.512 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-05T22:19:01.763 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-05T22:19:01.786 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-05T22:19:01.989 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-05T22:19:02.129 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-05T22:19:02.232 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-05T22:19:02.358 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-05T22:19:02.397 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-05T22:19:02.523 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:02.634 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-05T22:19:02.760 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-05T22:19:02.823 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-idna (3.3-1) ... 2023-12-05T22:19:03.053 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-05T22:19:03.141 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-05T22:19:03.354 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-05T22:19:03.597 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-05T22:19:03.706 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-05T22:19:03.832 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-05T22:19:03.921 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-05T22:19:04.150 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-05T22:19:04.277 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-05T22:19:04.355 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-05T22:19:04.598 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:19:04.661 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-05T22:19:04.732 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-05T22:19:04.921 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-05T22:19:05.038 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-05T22:19:05.109 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-05T22:19:05.209 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-05T22:19:05.281 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:19:05.310 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-05T22:19:05.390 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-05T22:19:05.587 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-05T22:19:05.724 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-05T22:19:05.855 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-05T22:19:06.034 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-05T22:19:06.114 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:19:06.240 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-05T22:19:06.310 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-05T22:19:06.436 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-05T22:19:06.558 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-05T22:19:06.712 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-05T22:19:06.833 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-05T22:19:06.972 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-05T22:19:07.085 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-05T22:19:07.300 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-05T22:19:07.327 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-05T22:19:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-05T22:19:07.677 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-05T22:19:07.829 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-05T22:19:07.953 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-05T22:19:07.955 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-05T22:19:08.249 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-05T22:19:08.297 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-05T22:19:08.374 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-05T22:19:08.398 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librados2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:08.499 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libsqlite3-mod-ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:08.625 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-05T22:19:08.667 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-05T22:19:08.961 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:09.001 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-05T22:19:09.096 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libradosstriper1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:09.222 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-05T22:19:09.260 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-05T22:19:09.361 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-05T22:19:09.462 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-05T22:19:09.481 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librbd1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:09.561 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-05T22:19:09.607 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-modules-core (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:09.733 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:09.797 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-05T22:19:09.932 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-05T22:19:09.958 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-05T22:19:09.958 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-05T22:19:10.341 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-05T22:19:10.518 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs-dev (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:10.609 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-05T22:19:10.644 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rados (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:10.710 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-05T22:19:10.770 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librgw2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:10.881 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-05T22:19:10.952 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-05T22:19:11.053 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-05T22:19:11.388 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-05T22:19:11.689 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-05T22:19:11.982 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-05T22:19:12.108 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-05T22:19:12.118 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rbd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:12.244 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up rbd-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:12.370 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:12.392 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-05T22:19:12.521 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cephfs (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:12.646 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:12.685 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-05T22:19:13.030 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-05T22:19:13.089 INFO:teuthology.orchestra.run.smithi190.stdout:Adding group ceph....done 2023-12-05T22:19:13.423 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-05T22:19:13.458 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user ceph....done 2023-12-05T22:19:13.634 INFO:teuthology.orchestra.run.smithi190.stdout:Setting system user ceph properties....done 2023-12-05T22:19:13.650 INFO:teuthology.orchestra.run.smithi190.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-05T22:19:13.707 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-05T22:19:13.784 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-05T22:19:14.051 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-05T22:19:14.130 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-05T22:19:14.152 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librados2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:14.278 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libsqlite3-mod-ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:14.396 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-05T22:19:14.673 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-test (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:14.732 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:14.799 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up radosgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:14.858 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libradosstriper1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:14.984 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-05T22:19:15.261 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librbd1 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:15.312 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-05T22:19:15.312 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-05T22:19:15.379 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-modules-core (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:15.513 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:15.741 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-05T22:19:15.741 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-05T22:19:15.842 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-base (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:16.186 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-05T22:19:16.273 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs-dev (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:16.398 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rados (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:16.524 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librgw2 (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:16.650 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-05T22:19:16.735 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mds (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:16.966 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-05T22:19:16.966 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-05T22:19:17.445 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up rbd-mirror (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:17.655 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-05T22:19:17.656 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-05T22:19:17.920 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rbd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:18.055 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up rbd-fuse (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:18.147 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:18.181 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:18.307 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cephfs (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:18.354 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-05T22:19:18.355 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-05T22:19:18.433 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-common (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:18.873 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-osd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:18.925 INFO:teuthology.orchestra.run.smithi062.stdout:Adding group ceph....done 2023-12-05T22:19:19.200 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-05T22:19:19.200 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-05T22:19:19.302 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user ceph....done 2023-12-05T22:19:19.464 INFO:teuthology.orchestra.run.smithi062.stdout:Setting system user ceph properties....done 2023-12-05T22:19:19.482 INFO:teuthology.orchestra.run.smithi062.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-05T22:19:19.637 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-05T22:19:19.767 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-k8sevents (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:19.868 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:19.994 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mon (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:20.019 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-05T22:19:20.208 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-05T22:19:20.208 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-05T22:19:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-test (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:20.693 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up radosgw (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:20.779 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:20.947 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:21.098 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-dashboard (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:21.168 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-05T22:19:21.168 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-05T22:19:21.257 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-volume (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:21.587 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-05T22:19:21.669 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-base (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:21.742 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-05T22:19:22.017 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-05T22:19:22.090 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-05T22:19:22.590 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mds (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:22.836 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-05T22:19:22.836 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-05T22:19:23.163 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-05T22:19:23.365 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up rbd-mirror (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:23.582 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-05T22:19:23.582 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-05T22:19:23.786 INFO:teuthology.orchestra.run.smithi190.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-05T22:19:23.789 DEBUG:teuthology.parallel:result is None 2023-12-05T22:19:24.100 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:24.330 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-05T22:19:24.330 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-05T22:19:24.851 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-osd (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:25.151 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-05T22:19:25.151 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-05T22:19:25.948 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-k8sevents (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:26.074 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:26.291 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mon (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:26.588 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-05T22:19:26.589 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-05T22:19:27.101 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-cephadm (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:27.227 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:27.328 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-dashboard (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:27.463 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-volume (19.0.0-21-g4a50e96e-1jammy) ... 2023-12-05T22:19:27.815 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-05T22:19:27.933 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-05T22:19:28.085 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-05T22:19:29.153 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-05T22:19:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-05T22:19:29.867 DEBUG:teuthology.parallel:result is None 2023-12-05T22:19:29.867 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4a50e96e7d9d85da28ef008617cb8342574bfd1b 2023-12-05T22:19:30.019 DEBUG:teuthology.orchestra.run.smithi062:> dpkg-query -W -f '${Version}' ceph 2023-12-05T22:19:30.039 INFO:teuthology.orchestra.run.smithi062.stdout:19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:19:30.039 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:19:30.039 INFO:teuthology.task.install:The correct ceph version 19.0.0-21-g4a50e96e-1jammy is installed. 2023-12-05T22:19:30.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4a50e96e7d9d85da28ef008617cb8342574bfd1b 2023-12-05T22:19:30.253 DEBUG:teuthology.orchestra.run.smithi190:> dpkg-query -W -f '${Version}' ceph 2023-12-05T22:19:30.272 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:19:30.273 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-21-g4a50e96e-1jammy 2023-12-05T22:19:30.273 INFO:teuthology.task.install:The correct ceph version 19.0.0-21-g4a50e96e-1jammy is installed. 2023-12-05T22:19:30.273 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-05T22:19:30.273 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:30.273 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-05T22:19:30.288 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:30.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-05T22:19:30.327 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-05T22:19:30.327 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:30.327 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2023-12-05T22:19:30.348 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-05T22:19:30.406 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:30.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2023-12-05T22:19:30.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-05T22:19:30.474 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-05T22:19:30.474 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:30.474 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-05T22:19:30.487 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-05T22:19:30.544 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:30.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-05T22:19:30.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-05T22:19:30.614 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-05T22:19:30.615 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:30.615 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2023-12-05T22:19:30.628 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-05T22:19:30.687 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:30.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2023-12-05T22:19:30.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-05T22:19:30.754 INFO:teuthology.run_tasks:Running task ceph... 2023-12-05T22:19:30.830 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-05T22:19:30.831 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2023-12-05T22:19:30.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2023-12-05T22:19:30.847 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-05T22:19:30.847 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-05T22:19:30.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-05T22:19:30.902 INFO:tasks.ceph:Creating extra log directories... 2023-12-05T22:19:30.902 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-05T22:19:30.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-05T22:19:30.971 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-05T22:19:30.972 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-05T22:19:30.972 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7478989', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2777, 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b', 'sleep_before_teardown': 0, 'subset': '17/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b', 'targets': {'smithi062.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKqroaZT7HdYX62YLD5DEPqGdvn/11IHy8kmg0TszQ4KFHbms/zi4XIFQ/5ysjy8atJOWH5R6XjS7bmPmFzLnsc=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWBgB238hKlGR5M5K13I6ZaYPHPvcklmz6GAjCQypqGTttuc6O4NwhdgZf/28VITHQriAuy2J0LPslmczqnfYo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-05_21:30:10', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-05T22:19:30.972 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-05T22:19:31.003 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-05T22:19:31.011 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2023-12-05T22:19:31.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2023-12-05T22:19:31.067 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:31.067 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2023-12-05T22:19:31.112 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-05T22:19:31.112 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2023-12-05T22:19:31.160 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-05T22:19:31.160 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:19:31.161 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 883 Links: 1 2023-12-05T22:19:31.161 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:19:31.161 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-12-05 22:14:05.762945339 +0000 2023-12-05T22:19:31.161 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-12-05 22:14:05.374948805 +0000 2023-12-05T22:19:31.161 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-12-05 22:14:05.374948805 +0000 2023-12-05T22:19:31.161 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-12-05T22:19:31.161 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-05T22:19:31.215 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-12-05T22:19:31.215 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-12-05T22:19:31.215 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000430415 s, 1.2 MB/s 2023-12-05T22:19:31.216 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-05T22:19:31.266 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2023-12-05T22:19:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-05T22:19:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:19:31.312 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-12-05T22:19:31.313 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:19:31.313 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-12-05 22:14:06.002943195 +0000 2023-12-05T22:19:31.313 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-12-05 22:14:05.998943231 +0000 2023-12-05T22:19:31.313 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-12-05 22:14:05.998943231 +0000 2023-12-05T22:19:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-12-05T22:19:31.313 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-05T22:19:31.367 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-12-05T22:19:31.367 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-12-05T22:19:31.367 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00041837 s, 1.2 MB/s 2023-12-05T22:19:31.368 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-05T22:19:31.418 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 920 Links: 1 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-12-05 22:14:06.982934441 +0000 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-12-05 22:14:06.610937763 +0000 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-12-05 22:14:06.610937763 +0000 2023-12-05T22:19:31.464 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-12-05T22:19:31.465 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-05T22:19:31.518 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-12-05T22:19:31.519 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-12-05T22:19:31.519 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000417428 s, 1.2 MB/s 2023-12-05T22:19:31.520 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-05T22:19:31.570 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 929 Links: 1 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-12-05 22:14:07.634928617 +0000 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-12-05 22:14:07.262931939 +0000 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-12-05 22:14:07.262931939 +0000 2023-12-05T22:19:31.619 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-12-05T22:19:31.619 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-05T22:19:31.671 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-12-05T22:19:31.671 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-12-05T22:19:31.671 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000318844 s, 1.6 MB/s 2023-12-05T22:19:31.672 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-05T22:19:31.722 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:19:31.722 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-05T22:19:31.722 INFO:tasks.ceph:Generating config... 2023-12-05T22:19:31.724 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-05T22:19:31.724 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-05T22:19:31.724 INFO:tasks.ceph:[client] log to stderr = False 2023-12-05T22:19:31.725 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-05T22:19:31.725 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-05T22:19:31.725 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:31.725 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:31.725 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:31.725 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:31.725 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:31.726 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:31.726 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:31.726 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:31.726 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-05T22:19:31.726 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-05T22:19:31.726 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-05T22:19:31.726 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-05T22:19:31.727 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-05T22:19:31.727 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-05T22:19:31.727 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-05T22:19:31.727 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-05T22:19:31.727 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-05T22:19:31.727 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-05T22:19:31.727 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-05T22:19:31.728 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-05T22:19:31.728 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-05T22:19:31.728 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-05T22:19:31.728 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-05T22:19:31.728 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-05T22:19:31.728 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-05T22:19:31.728 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-05T22:19:31.729 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-05T22:19:31.729 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-05T22:19:31.729 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-05T22:19:31.729 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-12-05T22:19:31.729 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-05T22:19:31.730 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-05T22:19:31.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-05T22:19:31.796 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/cluster1.keyring 2023-12-05T22:19:31.804 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-05T22:19:31.881 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-05T22:19:31.936 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.62')] 2023-12-05T22:19:31.937 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.62', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-05T22:19:31.938 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:31.938 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-05T22:19:31.984 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.62 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: generated fsid c638f251-f7eb-46b0-9628-a265a578cb3d 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = pacific 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 0 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:fsid c638f251-f7eb-46b0-9628-a265a578cb3d 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:last_changed 2023-12-05T22:19:32.066562+0000 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:created 2023-12-05T22:19:32.066562+0000 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:min_mon_release 16 (pacific) 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:election_strategy: 1 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2023-12-05T22:19:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-05T22:19:32.066 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-05T22:19:32.113 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID c638f251-f7eb-46b0-9628-a265a578cb3d... 2023-12-05T22:19:32.114 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-05T22:19:32.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-05T22:19:32.190 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2023-12-05T22:19:32.191 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2023-12-05T22:19:32.192 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2023-12-05T22:19:32.193 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = 172.21.15.62 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: mon client directed command retry = 5 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on pool no app = False 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject socket failures = 5000 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = c638f251-f7eb-46b0-9628-a265a578cb3d 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2023-12-05T22:19:32.194 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug trim objects = true 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.195 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-12-05T22:19:32.196 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: osd memory target = 1610612736 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:19:32.197 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-05T22:19:32.198 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2023-12-05T22:19:32.199 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd = 20 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd_mirror = 15 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: log to stderr = False 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: rbd default features = 125 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: rbd mirror image policy type = simple 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.0] 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.1] 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.2] 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.3] 2023-12-05T22:19:32.201 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.201 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.201 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2023-12-05T22:19:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2023-12-05T22:19:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-05T22:19:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = 172.21.15.62 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 5000 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = c638f251-f7eb-46b0-9628-a265a578cb3d 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2023-12-05T22:19:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2023-12-05T22:19:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: bdev async discard = True 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: bdev enable discard = True 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2023-12-05T22:19:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 1/20 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 1/20 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 4/10 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: osd memory target = 1610612736 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2023-12-05T22:19:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2023-12-05T22:19:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2023-12-05T22:19:32.209 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd = 20 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd_mirror = 15 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: log to stderr = False 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: rbd default features = 125 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: rbd mirror image policy type = simple 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2023-12-05T22:19:32.210 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.211 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.211 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2023-12-05T22:19:32.211 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.211 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.211 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2023-12-05T22:19:32.211 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:19:32.211 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:19:32.212 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-05T22:19:32.212 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-05T22:19:32.305 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-05T22:19:32.305 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:32.305 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-05T22:19:32.310 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:32.310 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-05T22:19:32.356 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T22:19:32.356 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:32.356 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-05T22:19:32.356 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-05T22:19:32.421 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:32.421 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:32.468 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T22:19:32.468 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:32.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-05T22:19:32.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-05T22:19:32.491 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:32.492 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:32.539 INFO:tasks.ceph:Setting up mon nodes... 2023-12-05T22:19:32.539 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-05T22:19:32.539 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-05T22:19:32.582 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-05T22:19:32.585 INFO:tasks.ceph:Setting up mds nodes... 2023-12-05T22:19:32.585 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-05T22:19:32.585 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-05T22:19:32.657 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-05T22:19:32.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-05T22:19:32.702 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-05T22:19:32.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-05T22:19:32.812 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-05T22:19:32.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-05T22:19:32.898 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-05T22:19:32.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-05T22:19:32.981 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-05T22:19:32.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-05T22:19:33.069 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-05T22:19:33.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-05T22:19:33.157 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-05T22:19:33.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-05T22:19:33.243 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-05T22:19:33.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-05T22:19:33.323 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-05T22:19:33.334 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-05T22:19:33.334 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-05T22:19:33.334 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-05T22:19:33.349 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:19:33.349 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-05T22:19:33.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T22:19:33.349 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-05T22:19:33.416 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-05T22:19:33.416 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-05T22:19:33.416 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-05T22:19:33.416 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-05T22:19:33.416 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-05T22:19:33.417 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2023-12-05T22:19:33.417 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-05T22:19:33.417 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-05T22:19:33.417 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-05T22:19:33.417 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-05T22:19:33.875 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2023-12-05T22:19:33.876 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2023-12-05T22:19:33.877 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-05T22:19:33.971 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-05T22:19:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:19:33.984 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2023-12-05T22:19:33.984 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-05T22:19:34.040 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:19:34.040 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-05T22:19:34.040 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T22:19:34.040 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-05T22:19:34.105 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-05T22:19:34.105 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-05T22:19:34.105 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-05T22:19:34.105 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-05T22:19:34.106 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-05T22:19:34.106 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2023-12-05T22:19:34.106 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-05T22:19:34.106 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-05T22:19:34.106 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-05T22:19:34.106 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-05T22:19:34.572 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2023-12-05T22:19:34.573 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2023-12-05T22:19:34.573 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-05T22:19:34.727 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-05T22:19:34.738 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2023-12-05T22:19:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:19:34.739 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-05T22:19:34.792 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:19:34.792 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-05T22:19:34.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T22:19:34.792 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-05T22:19:34.856 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-05T22:19:34.856 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-05T22:19:34.856 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-05T22:19:34.856 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-05T22:19:34.856 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-05T22:19:34.856 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2023-12-05T22:19:34.857 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-05T22:19:34.857 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-05T22:19:34.857 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-05T22:19:34.857 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-05T22:19:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2023-12-05T22:19:35.345 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2023-12-05T22:19:35.345 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-05T22:19:35.476 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-05T22:19:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:19:35.486 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2023-12-05T22:19:35.487 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:35.590 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:35.587+0000 7fea97e08640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-05T22:19:35.591 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:35.587+0000 7fea97e08640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-05T22:19:35.591 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:35.587+0000 7fea97e08640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-05T22:19:35.591 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:35.587+0000 7fea97e08640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-05T22:19:35.591 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:35.587+0000 7fea97e08640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-05T22:19:35.592 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:35.591+0000 7fea97e08640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-05T22:19:38.597 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-05T22:19:38.611 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:38.697 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:38.695+0000 7f895d4f0640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-05T22:19:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:38.695+0000 7f895d4f0640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-05T22:19:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:38.695+0000 7f895d4f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-05T22:19:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:38.695+0000 7f895d4f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-05T22:19:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:38.695+0000 7f895d4f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-05T22:19:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:38.695+0000 7f895d4f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-05T22:19:41.723 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-05T22:19:41.781 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:41.864 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:41.863+0000 7f2397c0e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-05T22:19:41.865 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:41.863+0000 7f2397c0e640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-05T22:19:41.865 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:41.863+0000 7f2397c0e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-05T22:19:41.865 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:41.863+0000 7f2397c0e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-05T22:19:41.865 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:41.863+0000 7f2397c0e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-05T22:19:41.865 INFO:teuthology.orchestra.run.smithi062.stderr:2023-12-05T22:19:41.863+0000 7f2397c0e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-05T22:19:44.899 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-05T22:19:44.957 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-05T22:19:44.957 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:44.957 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-05T22:19:45.012 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:45.012 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-05T22:19:45.068 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:45.068 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-05T22:19:45.125 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:45.125 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-05T22:19:45.180 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:45.180 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-05T22:19:45.228 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.228 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-05T22:19:45.234 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.234 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-05T22:19:45.284 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.284 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-05T22:19:45.332 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.332 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-05T22:19:45.378 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.379 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-05T22:19:45.427 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.428 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-05T22:19:45.474 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.475 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-05T22:19:45.522 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:19:45.522 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-05T22:19:45.571 INFO:tasks.ceph:Adding keys to all mons... 2023-12-05T22:19:45.571 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-05T22:19:45.583 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2023-12-05T22:19:45.583 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD0oW9lrLjYIhAALWK4PHK6/rHPBT3yfV6pLw== 2023-12-05T22:19:45.583 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2023-12-05T22:19:45.583 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD3oW9lhuBUIxAAQ3lNy704BOVBhiqlI9t8jA== 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD6oW9lNFWvKRAAW/14J0Ph9f2YDTKI0Ctwig== 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD9oW9l862hMxAA9sALeHhYw1MRsieaE9ZoBA== 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD0oW9lgGJQJxAADTTFnM+uuuf+BHiCyHItGA== 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror] 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD0oW9lJCsZKhAAxLm9g9Qh1U0sEwopzaoReQ== 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.0] 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD0oW9lj2+eMBAA4DRCV3jej7AlbiyvnT9pLQ== 2023-12-05T22:19:45.584 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.1] 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD0oW9lLN2+NRAARJkTn4N4FjagvgrVbgJiRw== 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.2] 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD0oW9l0Qe7OhAAO8Uwj35ORjG7jGz8G8gqbg== 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.3] 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD1oW9l78tWBBAAHQRjRh4X+jPT166GyGPHHw== 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.4] 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD1oW9lcySbCRAA2BrsEzC63hjdTN7GsrYTbA== 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.5] 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD1oW9lDqu9DhAANOZtN2yswOn7JT1Xbvp6JQ== 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.6] 2023-12-05T22:19:45.585 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQD1oW9loQ97ExAA54xGnqmUG8YxEVXoZQsBjQ== 2023-12-05T22:19:45.586 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-05T22:19:45.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-05T22:19:45.717 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-05T22:19:45.804 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-05T22:19:45.845 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:45.930 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.015 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.102 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.149 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.192 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.277 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.368 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.414 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:19:46.499 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-05T22:19:46.499 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-05T22:19:46.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-05T22:19:46.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-05T22:19:46.656 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-05T22:19:46.704 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-05T22:19:46.704 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-05T22:19:46.704 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-05T22:19:46.749 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-05T22:19:46.749 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-05T22:19:46.749 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-05T22:19:46.749 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-05T22:19:46.751 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-05T22:19:46.751 INFO:tasks.ceph:Setting crush tunables to default 2023-12-05T22:19:46.752 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-05T22:19:47.002 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2023-12-05T22:19:47.020 INFO:tasks.ceph:check_enable_crimson: False 2023-12-05T22:19:47.020 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-05T22:19:47.020 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:47.020 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-05T22:19:47.031 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:47.031 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-05T22:19:47.085 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:19:47.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-05T22:19:47.118 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:47.115+0000 7fd142745180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-05T22:19:47.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd new 15e3053e-fa26-40a9-958f-4e09c36e0052 0 2023-12-05T22:19:47.375 INFO:teuthology.orchestra.run.smithi062.stdout:0 2023-12-05T22:19:47.393 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd new 92c57430-c95d-44c9-887f-2f80c09752b7 1 2023-12-05T22:19:47.599 INFO:teuthology.orchestra.run.smithi062.stdout:1 2023-12-05T22:19:47.617 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd new 641eb594-1137-4504-a975-626c541f33bc 2 2023-12-05T22:19:47.820 INFO:teuthology.orchestra.run.smithi062.stdout:2 2023-12-05T22:19:47.838 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-05T22:19:47.838 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-05T22:19:47.841 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-05T22:19:47.841 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-05T22:19:47.841 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-05T22:19:47.843 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-05T22:19:47.843 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-05T22:19:47.843 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-05T22:19:47.845 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-05T22:19:47.846 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-05T22:19:47.934 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:47.931+0000 7fd142745180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-05T22:19:48.032 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:19:48.032 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"fsid":"c638f251-f7eb-46b0-9628-a265a578cb3d","created":"2023-12-05T22:19:46.901568+0000","modified":"2023-12-05T22:19:47.820370+0000","last_up_change":"0.000000","last_in_change":"2023-12-05T22:19:47.820370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15e3053e-fa26-40a9-958f-4e09c36e0052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"92c57430-c95d-44c9-887f-2f80c09752b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"641eb594-1137-4504-a975-626c541f33bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:19:48.050 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-05T22:19:48.050 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-05T22:19:48.138 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:48.135+0000 7fd142745180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-05T22:19:48.352 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-05T22:19:48.461 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:19:48.459+0000 7efd704bc640 -1 Falling back to public interface 2023-12-05T22:19:48.477 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:19:48.459+0000 7f4b1e459640 -1 Falling back to public interface 2023-12-05T22:19:48.484 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:19:48.483+0000 7f1629e2c640 -1 Falling back to public interface 2023-12-05T22:19:48.538 INFO:teuthology.misc.health.smithi062.stdout: 2023-12-05T22:19:48.538 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":5,"fsid":"c638f251-f7eb-46b0-9628-a265a578cb3d","created":"2023-12-05T22:19:46.901568+0000","modified":"2023-12-05T22:19:47.820370+0000","last_up_change":"0.000000","last_in_change":"2023-12-05T22:19:47.820370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15e3053e-fa26-40a9-958f-4e09c36e0052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"92c57430-c95d-44c9-887f-2f80c09752b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"641eb594-1137-4504-a975-626c541f33bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:19:48.552 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-05T22:19:48.755 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:48.755+0000 7fd142745180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-05T22:19:48.812 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:48.811+0000 7fd142745180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-05T22:19:48.867 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:48.863+0000 7fd142745180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-05T22:19:48.948 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:48.947+0000 7fd142745180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-05T22:19:49.002 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:48.999+0000 7fd142745180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-05T22:19:49.083 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:49.079+0000 7fd142745180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-05T22:19:49.139 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:49.135+0000 7fd142745180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-05T22:19:49.285 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:49.283+0000 7fd142745180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-05T22:19:49.347 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:49.347+0000 7fd142745180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-05T22:19:50.066 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-05T22:19:50.066 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-05T22:19:50.066 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-12-05T22:19:50.143 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:50.139+0000 7fd142745180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-05T22:19:50.302 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:50.299+0000 7fd142745180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-05T22:19:50.585 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:50.583+0000 7fd142745180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-05T22:19:50.666 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:50.663+0000 7fd142745180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-05T22:19:50.805 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:50.803+0000 7fd142745180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-05T22:19:51.608 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:51.607+0000 7fd142745180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-05T22:19:51.687 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:51.683+0000 7fd142745180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-05T22:19:51.867 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:51.863+0000 7fd142745180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-05T22:19:51.944 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:51.943+0000 7fd142745180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-05T22:19:52.250 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:52.247+0000 7fd142745180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-05T22:19:52.287 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:19:52.283+0000 7f1629e2c640 -1 osd.2 0 log_to_monitors true 2023-12-05T22:19:52.300 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:19:52.299+0000 7f4b1e459640 -1 osd.1 0 log_to_monitors true 2023-12-05T22:19:52.324 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:52.323+0000 7fd142745180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-05T22:19:52.350 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:19:52.347+0000 7efd704bc640 -1 osd.0 0 log_to_monitors true 2023-12-05T22:19:53.560 INFO:tasks.ceph.cluster1.mgr.x.smithi062.stderr:2023-12-05T22:19:53.559+0000 7fd0e9eb9640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-05T22:19:54.856 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-05T22:19:55.307 INFO:teuthology.misc.health.smithi062.stdout: 2023-12-05T22:19:55.308 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":7,"fsid":"c638f251-f7eb-46b0-9628-a265a578cb3d","created":"2023-12-05T22:19:46.901568+0000","modified":"2023-12-05T22:19:53.844169+0000","last_up_change":"0.000000","last_in_change":"2023-12-05T22:19:47.820370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"15e3053e-fa26-40a9-958f-4e09c36e0052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"92c57430-c95d-44c9-887f-2f80c09752b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"641eb594-1137-4504-a975-626c541f33bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:19:55.333 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-05T22:19:55.840 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:19:55.839+0000 7f16255b0640 -1 osd.2 0 waiting for initial osdmap 2023-12-05T22:19:55.868 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:19:55.867+0000 7efd6bc40640 -1 osd.0 0 waiting for initial osdmap 2023-12-05T22:19:55.872 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:19:55.871+0000 7f4b19bdd640 -1 osd.1 0 waiting for initial osdmap 2023-12-05T22:19:55.876 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:19:55.875+0000 7f161dbf7640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-05T22:19:55.896 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:19:55.895+0000 7efd64287640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-05T22:19:55.900 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:19:55.899+0000 7f4b12224640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-05T22:20:01.636 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-05T22:20:01.933 INFO:teuthology.misc.health.smithi062.stdout: 2023-12-05T22:20:01.934 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":12,"fsid":"c638f251-f7eb-46b0-9628-a265a578cb3d","created":"2023-12-05T22:19:46.901568+0000","modified":"2023-12-05T22:20:00.904529+0000","last_up_change":"2023-12-05T22:19:56.877430+0000","last_in_change":"2023-12-05T22:19:47.820370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-05T22:19:58.427473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"15e3053e-fa26-40a9-958f-4e09c36e0052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6805","nonce":3557713586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6811","nonce":3557713586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6815","nonce":3557713586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6813","nonce":3557713586}]},"public_addr":"172.21.15.62:6805/3557713586","cluster_addr":"172.21.15.62:6811/3557713586","heartbeat_back_addr":"172.21.15.62:6815/3557713586","heartbeat_front_addr":"172.21.15.62:6813/3557713586","state":["exists","up"]},{"osd":1,"uuid":"92c57430-c95d-44c9-887f-2f80c09752b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6801","nonce":203106146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6803","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6804","nonce":203106146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6810","nonce":203106146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6807","nonce":203106146}]},"public_addr":"172.21.15.62:6801/203106146","cluster_addr":"172.21.15.62:6804/203106146","heartbeat_back_addr":"172.21.15.62:6810/203106146","heartbeat_front_addr":"172.21.15.62:6807/203106146","state":["exists","up"]},{"osd":2,"uuid":"641eb594-1137-4504-a975-626c541f33bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6817","nonce":3043463782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6819","nonce":3043463782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6823","nonce":3043463782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6821","nonce":3043463782}]},"public_addr":"172.21.15.62:6817/3043463782","cluster_addr":"172.21.15.62:6819/3043463782","heartbeat_back_addr":"172.21.15.62:6823/3043463782","heartbeat_front_addr":"172.21.15.62:6821/3043463782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.334766+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.341700+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.291465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:01.951 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-05T22:20:01.951 INFO:tasks.ceph:Creating RBD pool 2023-12-05T22:20:01.951 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-05T22:20:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'rbd' created 2023-12-05T22:20:02.944 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-05T22:20:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-05T22:20:03.957 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-05T22:20:03.958 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-05T22:20:03.958 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-05T22:20:03.958 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-05T22:20:03.959 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-05T22:20:04.297 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:04.316 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":1846979903},{"type":"v1","addr":"172.21.15.62:6825","nonce":1846979903}]},"active_addr":"172.21.15.62:6825/1846979903","active_change":"2023-12-05T22:19:52.346748+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2401666475}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":875022638}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2769088028}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":3766562265}]}]} 2023-12-05T22:20:04.320 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-05T22:20:04.321 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-05T22:20:04.321 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-05T22:20:04.621 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:04.622 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":14,"fsid":"c638f251-f7eb-46b0-9628-a265a578cb3d","created":"2023-12-05T22:19:46.901568+0000","modified":"2023-12-05T22:20:03.920374+0000","last_up_change":"2023-12-05T22:19:56.877430+0000","last_in_change":"2023-12-05T22:19:47.820370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-05T22:19:58.427473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-05T22:20:02.290443+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"15e3053e-fa26-40a9-958f-4e09c36e0052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6805","nonce":3557713586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6811","nonce":3557713586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6815","nonce":3557713586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6813","nonce":3557713586}]},"public_addr":"172.21.15.62:6805/3557713586","cluster_addr":"172.21.15.62:6811/3557713586","heartbeat_back_addr":"172.21.15.62:6815/3557713586","heartbeat_front_addr":"172.21.15.62:6813/3557713586","state":["exists","up"]},{"osd":1,"uuid":"92c57430-c95d-44c9-887f-2f80c09752b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6801","nonce":203106146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6803","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6804","nonce":203106146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6810","nonce":203106146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6807","nonce":203106146}]},"public_addr":"172.21.15.62:6801/203106146","cluster_addr":"172.21.15.62:6804/203106146","heartbeat_back_addr":"172.21.15.62:6810/203106146","heartbeat_front_addr":"172.21.15.62:6807/203106146","state":["exists","up"]},{"osd":2,"uuid":"641eb594-1137-4504-a975-626c541f33bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6817","nonce":3043463782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6819","nonce":3043463782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6823","nonce":3043463782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6821","nonce":3043463782}]},"public_addr":"172.21.15.62:6817/3043463782","cluster_addr":"172.21.15.62:6819/3043463782","heartbeat_back_addr":"172.21.15.62:6823/3043463782","heartbeat_front_addr":"172.21.15.62:6821/3043463782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.334766+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.341700+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.291465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:04.640 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-05T22:20:04.640 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-05T22:20:04.935 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:04.935 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":14,"fsid":"c638f251-f7eb-46b0-9628-a265a578cb3d","created":"2023-12-05T22:19:46.901568+0000","modified":"2023-12-05T22:20:03.920374+0000","last_up_change":"2023-12-05T22:19:56.877430+0000","last_in_change":"2023-12-05T22:19:47.820370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-05T22:19:58.427473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-05T22:20:02.290443+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"15e3053e-fa26-40a9-958f-4e09c36e0052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6805","nonce":3557713586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6811","nonce":3557713586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6815","nonce":3557713586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6813","nonce":3557713586}]},"public_addr":"172.21.15.62:6805/3557713586","cluster_addr":"172.21.15.62:6811/3557713586","heartbeat_back_addr":"172.21.15.62:6815/3557713586","heartbeat_front_addr":"172.21.15.62:6813/3557713586","state":["exists","up"]},{"osd":1,"uuid":"92c57430-c95d-44c9-887f-2f80c09752b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6801","nonce":203106146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6803","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6804","nonce":203106146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6810","nonce":203106146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6807","nonce":203106146}]},"public_addr":"172.21.15.62:6801/203106146","cluster_addr":"172.21.15.62:6804/203106146","heartbeat_back_addr":"172.21.15.62:6810/203106146","heartbeat_front_addr":"172.21.15.62:6807/203106146","state":["exists","up"]},{"osd":2,"uuid":"641eb594-1137-4504-a975-626c541f33bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6817","nonce":3043463782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6819","nonce":3043463782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6823","nonce":3043463782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6821","nonce":3043463782}]},"public_addr":"172.21.15.62:6817/3043463782","cluster_addr":"172.21.15.62:6819/3043463782","heartbeat_back_addr":"172.21.15.62:6823/3043463782","heartbeat_front_addr":"172.21.15.62:6821/3043463782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.334766+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.341700+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.291465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:04.953 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-05T22:20:04.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-05T22:20:04.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-05T22:20:05.105 INFO:teuthology.orchestra.run.smithi062.stdout:34359738374 2023-12-05T22:20:05.106 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-05T22:20:05.121 INFO:teuthology.orchestra.run.smithi062.stdout:34359738375 2023-12-05T22:20:05.122 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-05T22:20:05.122 INFO:teuthology.orchestra.run.smithi062.stdout:34359738373 2023-12-05T22:20:05.123 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-05T22:20:05.409 INFO:teuthology.orchestra.run.smithi062.stdout:34359738373 2023-12-05T22:20:05.427 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.2 2023-12-05T22:20:05.451 INFO:teuthology.orchestra.run.smithi062.stdout:34359738372 2023-12-05T22:20:05.464 INFO:teuthology.orchestra.run.smithi062.stdout:34359738374 2023-12-05T22:20:05.468 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2023-12-05T22:20:05.482 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738374 for osd.0 2023-12-05T22:20:06.428 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-05T22:20:06.469 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-05T22:20:06.482 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-05T22:20:06.734 INFO:teuthology.orchestra.run.smithi062.stdout:34359738373 2023-12-05T22:20:06.756 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.2 2023-12-05T22:20:06.782 INFO:teuthology.orchestra.run.smithi062.stdout:34359738372 2023-12-05T22:20:06.804 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2023-12-05T22:20:06.848 INFO:teuthology.orchestra.run.smithi062.stdout:34359738374 2023-12-05T22:20:06.866 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738374 for osd.0 2023-12-05T22:20:07.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-05T22:20:07.805 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-05T22:20:07.867 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-05T22:20:08.067 INFO:teuthology.orchestra.run.smithi062.stdout:34359738374 2023-12-05T22:20:08.084 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2023-12-05T22:20:08.084 DEBUG:teuthology.parallel:result is None 2023-12-05T22:20:08.133 INFO:teuthology.orchestra.run.smithi062.stdout:34359738373 2023-12-05T22:20:08.150 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2023-12-05T22:20:08.150 DEBUG:teuthology.parallel:result is None 2023-12-05T22:20:08.192 INFO:teuthology.orchestra.run.smithi062.stdout:34359738375 2023-12-05T22:20:08.209 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738375 for osd.0 2023-12-05T22:20:08.209 DEBUG:teuthology.parallel:result is None 2023-12-05T22:20:08.209 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-05T22:20:08.209 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-05T22:20:08.497 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:08.498 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2023-12-05T22:20:08.516 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-05T22:20:08.354075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82596,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032924,"statfs":{"total":289910292480,"available":289825714176,"internally_reserved":0,"allocated":1572864,"data_stored":1288888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.432041"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.309210+0000","last_change":"2023-12-05T22:20:03.947467+0000","last_active":"2023-12-05T22:20:06.309210+0000","last_peered":"2023-12-05T22:20:06.309210+0000","last_clean":"2023-12-05T22:20:06.309210+0000","last_became_active":"2023-12-05T22:20:03.947069+0000","last_became_peered":"2023-12-05T22:20:03.947069+0000","last_unstale":"2023-12-05T22:20:06.309210+0000","last_undegraded":"2023-12-05T22:20:06.309210+0000","last_fullsized":"2023-12-05T22:20:06.309210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:30:10.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.309214+0000","last_change":"2023-12-05T22:20:03.955027+0000","last_active":"2023-12-05T22:20:06.309214+0000","last_peered":"2023-12-05T22:20:06.309214+0000","last_clean":"2023-12-05T22:20:06.309214+0000","last_became_active":"2023-12-05T22:20:03.954510+0000","last_became_peered":"2023-12-05T22:20:03.954510+0000","last_unstale":"2023-12-05T22:20:06.309214+0000","last_undegraded":"2023-12-05T22:20:06.309214+0000","last_fullsized":"2023-12-05T22:20:06.309214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:23:12.821704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.309148+0000","last_change":"2023-12-05T22:20:03.955392+0000","last_active":"2023-12-05T22:20:06.309148+0000","last_peered":"2023-12-05T22:20:06.309148+0000","last_clean":"2023-12-05T22:20:06.309148+0000","last_became_active":"2023-12-05T22:20:03.954778+0000","last_became_peered":"2023-12-05T22:20:03.954778+0000","last_unstale":"2023-12-05T22:20:06.309148+0000","last_undegraded":"2023-12-05T22:20:06.309148+0000","last_fullsized":"2023-12-05T22:20:06.309148+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:49:04.013308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.308997+0000","last_change":"2023-12-05T22:20:03.954927+0000","last_active":"2023-12-05T22:20:06.308997+0000","last_peered":"2023-12-05T22:20:06.308997+0000","last_clean":"2023-12-05T22:20:06.308997+0000","last_became_active":"2023-12-05T22:20:03.954437+0000","last_became_peered":"2023-12-05T22:20:03.954437+0000","last_unstale":"2023-12-05T22:20:06.308997+0000","last_undegraded":"2023-12-05T22:20:06.308997+0000","last_fullsized":"2023-12-05T22:20:06.308997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:36:37.288695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:04.940805+0000","last_change":"2023-12-05T22:20:03.949404+0000","last_active":"2023-12-05T22:20:04.940805+0000","last_peered":"2023-12-05T22:20:04.940805+0000","last_clean":"2023-12-05T22:20:04.940805+0000","last_became_active":"2023-12-05T22:20:03.948733+0000","last_became_peered":"2023-12-05T22:20:03.948733+0000","last_unstale":"2023-12-05T22:20:04.940805+0000","last_undegraded":"2023-12-05T22:20:04.940805+0000","last_fullsized":"2023-12-05T22:20:04.940805+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:35:07.902401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:03.955284+0000","last_change":"2023-12-05T22:20:03.950953+0000","last_active":"2023-12-05T22:20:03.955284+0000","last_peered":"2023-12-05T22:20:03.955284+0000","last_clean":"2023-12-05T22:20:03.955284+0000","last_became_active":"2023-12-05T22:20:03.950488+0000","last_became_peered":"2023-12-05T22:20:03.950488+0000","last_unstale":"2023-12-05T22:20:03.955284+0000","last_undegraded":"2023-12-05T22:20:03.955284+0000","last_fullsized":"2023-12-05T22:20:03.955284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:01:32.522865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:03.950071+0000","last_change":"2023-12-05T22:20:03.948835+0000","last_active":"2023-12-05T22:20:03.950071+0000","last_peered":"2023-12-05T22:20:03.950071+0000","last_clean":"2023-12-05T22:20:03.950071+0000","last_became_active":"2023-12-05T22:20:03.948291+0000","last_became_peered":"2023-12-05T22:20:03.948291+0000","last_unstale":"2023-12-05T22:20:03.950071+0000","last_undegraded":"2023-12-05T22:20:03.950071+0000","last_fullsized":"2023-12-05T22:20:03.950071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:20:25.279318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.308987+0000","last_change":"2023-12-05T22:20:03.944949+0000","last_active":"2023-12-05T22:20:06.308987+0000","last_peered":"2023-12-05T22:20:06.308987+0000","last_clean":"2023-12-05T22:20:06.308987+0000","last_became_active":"2023-12-05T22:20:03.944294+0000","last_became_peered":"2023-12-05T22:20:03.944294+0000","last_unstale":"2023-12-05T22:20:06.308987+0000","last_undegraded":"2023-12-05T22:20:06.308987+0000","last_fullsized":"2023-12-05T22:20:06.308987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:52:50.330306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:03.943353+0000","last_change":"2023-12-05T22:20:00.475423+0000","last_active":"2023-12-05T22:20:03.943353+0000","last_peered":"2023-12-05T22:20:03.943353+0000","last_clean":"2023-12-05T22:20:03.943353+0000","last_became_active":"2023-12-05T22:20:00.473094+0000","last_became_peered":"2023-12-05T22:20:00.473094+0000","last_unstale":"2023-12-05T22:20:03.943353+0000","last_undegraded":"2023-12-05T22:20:03.943353+0000","last_fullsized":"2023-12-05T22:20:03.943353+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_clean_scrub_stamp":"2023-12-05T22:19:58.883597+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:32:18.798306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T22:20:08.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-05T22:20:08.825 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2023-12-05T22:20:08.844 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-05T22:20:08.354075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82596,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032924,"statfs":{"total":289910292480,"available":289825714176,"internally_reserved":0,"allocated":1572864,"data_stored":1288888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.432041"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.309210+0000","last_change":"2023-12-05T22:20:03.947467+0000","last_active":"2023-12-05T22:20:06.309210+0000","last_peered":"2023-12-05T22:20:06.309210+0000","last_clean":"2023-12-05T22:20:06.309210+0000","last_became_active":"2023-12-05T22:20:03.947069+0000","last_became_peered":"2023-12-05T22:20:03.947069+0000","last_unstale":"2023-12-05T22:20:06.309210+0000","last_undegraded":"2023-12-05T22:20:06.309210+0000","last_fullsized":"2023-12-05T22:20:06.309210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:30:10.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.309214+0000","last_change":"2023-12-05T22:20:03.955027+0000","last_active":"2023-12-05T22:20:06.309214+0000","last_peered":"2023-12-05T22:20:06.309214+0000","last_clean":"2023-12-05T22:20:06.309214+0000","last_became_active":"2023-12-05T22:20:03.954510+0000","last_became_peered":"2023-12-05T22:20:03.954510+0000","last_unstale":"2023-12-05T22:20:06.309214+0000","last_undegraded":"2023-12-05T22:20:06.309214+0000","last_fullsized":"2023-12-05T22:20:06.309214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:23:12.821704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.309148+0000","last_change":"2023-12-05T22:20:03.955392+0000","last_active":"2023-12-05T22:20:06.309148+0000","last_peered":"2023-12-05T22:20:06.309148+0000","last_clean":"2023-12-05T22:20:06.309148+0000","last_became_active":"2023-12-05T22:20:03.954778+0000","last_became_peered":"2023-12-05T22:20:03.954778+0000","last_unstale":"2023-12-05T22:20:06.309148+0000","last_undegraded":"2023-12-05T22:20:06.309148+0000","last_fullsized":"2023-12-05T22:20:06.309148+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:49:04.013308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.308997+0000","last_change":"2023-12-05T22:20:03.954927+0000","last_active":"2023-12-05T22:20:06.308997+0000","last_peered":"2023-12-05T22:20:06.308997+0000","last_clean":"2023-12-05T22:20:06.308997+0000","last_became_active":"2023-12-05T22:20:03.954437+0000","last_became_peered":"2023-12-05T22:20:03.954437+0000","last_unstale":"2023-12-05T22:20:06.308997+0000","last_undegraded":"2023-12-05T22:20:06.308997+0000","last_fullsized":"2023-12-05T22:20:06.308997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:36:37.288695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:04.940805+0000","last_change":"2023-12-05T22:20:03.949404+0000","last_active":"2023-12-05T22:20:04.940805+0000","last_peered":"2023-12-05T22:20:04.940805+0000","last_clean":"2023-12-05T22:20:04.940805+0000","last_became_active":"2023-12-05T22:20:03.948733+0000","last_became_peered":"2023-12-05T22:20:03.948733+0000","last_unstale":"2023-12-05T22:20:04.940805+0000","last_undegraded":"2023-12-05T22:20:04.940805+0000","last_fullsized":"2023-12-05T22:20:04.940805+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:35:07.902401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:03.955284+0000","last_change":"2023-12-05T22:20:03.950953+0000","last_active":"2023-12-05T22:20:03.955284+0000","last_peered":"2023-12-05T22:20:03.955284+0000","last_clean":"2023-12-05T22:20:03.955284+0000","last_became_active":"2023-12-05T22:20:03.950488+0000","last_became_peered":"2023-12-05T22:20:03.950488+0000","last_unstale":"2023-12-05T22:20:03.955284+0000","last_undegraded":"2023-12-05T22:20:03.955284+0000","last_fullsized":"2023-12-05T22:20:03.955284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:01:32.522865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:03.950071+0000","last_change":"2023-12-05T22:20:03.948835+0000","last_active":"2023-12-05T22:20:03.950071+0000","last_peered":"2023-12-05T22:20:03.950071+0000","last_clean":"2023-12-05T22:20:03.950071+0000","last_became_active":"2023-12-05T22:20:03.948291+0000","last_became_peered":"2023-12-05T22:20:03.948291+0000","last_unstale":"2023-12-05T22:20:03.950071+0000","last_undegraded":"2023-12-05T22:20:03.950071+0000","last_fullsized":"2023-12-05T22:20:03.950071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:20:25.279318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:06.308987+0000","last_change":"2023-12-05T22:20:03.944949+0000","last_active":"2023-12-05T22:20:06.308987+0000","last_peered":"2023-12-05T22:20:06.308987+0000","last_clean":"2023-12-05T22:20:06.308987+0000","last_became_active":"2023-12-05T22:20:03.944294+0000","last_became_peered":"2023-12-05T22:20:03.944294+0000","last_unstale":"2023-12-05T22:20:06.308987+0000","last_undegraded":"2023-12-05T22:20:06.308987+0000","last_fullsized":"2023-12-05T22:20:06.308987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:52:50.330306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:03.943353+0000","last_change":"2023-12-05T22:20:00.475423+0000","last_active":"2023-12-05T22:20:03.943353+0000","last_peered":"2023-12-05T22:20:03.943353+0000","last_clean":"2023-12-05T22:20:03.943353+0000","last_became_active":"2023-12-05T22:20:00.473094+0000","last_became_peered":"2023-12-05T22:20:00.473094+0000","last_unstale":"2023-12-05T22:20:03.943353+0000","last_undegraded":"2023-12-05T22:20:03.943353+0000","last_fullsized":"2023-12-05T22:20:03.943353+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_clean_scrub_stamp":"2023-12-05T22:19:58.883597+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:32:18.798306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T22:20:08.844 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-05T22:20:08.845 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-05T22:20:08.845 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-05T22:20:08.845 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-05T22:20:09.165 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:09.165 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-05T22:20:09.183 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-05T22:20:09.184 INFO:teuthology.run_tasks:Running task ceph... 2023-12-05T22:20:09.195 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-05T22:20:09.196 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-05T22:20:09.196 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7478989', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2777, 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b', 'sleep_before_teardown': 0, 'subset': '17/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b', 'targets': {'smithi062.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKqroaZT7HdYX62YLD5DEPqGdvn/11IHy8kmg0TszQ4KFHbms/zi4XIFQ/5ysjy8atJOWH5R6XjS7bmPmFzLnsc=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWBgB238hKlGR5M5K13I6ZaYPHPvcklmz6GAjCQypqGTttuc6O4NwhdgZf/28VITHQriAuy2J0LPslmczqnfYo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4a50e96e7d9d85da28ef008617cb8342574bfd1b'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-05_21:30:10', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-05T22:20:09.196 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-05T22:20:09.198 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-05T22:20:09.206 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2023-12-05T22:20:09.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2023-12-05T22:20:09.263 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:09.263 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2023-12-05T22:20:09.310 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-05T22:20:09.310 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2023-12-05T22:20:09.358 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-05T22:20:09.359 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:20:09.359 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 873 Links: 1 2023-12-05T22:20:09.359 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:20:09.359 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-12-05 22:14:05.040383637 +0000 2023-12-05T22:20:09.359 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-12-05 22:14:04.640387209 +0000 2023-12-05T22:20:09.359 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-12-05 22:14:04.640387209 +0000 2023-12-05T22:20:09.359 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-12-05T22:20:09.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-05T22:20:09.413 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-12-05T22:20:09.413 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-12-05T22:20:09.413 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00043768 s, 1.2 MB/s 2023-12-05T22:20:09.414 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-05T22:20:09.465 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 893 Links: 1 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-12-05 22:14:05.620378456 +0000 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-12-05 22:14:05.272381565 +0000 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-12-05 22:14:05.272381565 +0000 2023-12-05T22:20:09.510 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-12-05T22:20:09.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-05T22:20:09.564 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-12-05T22:20:09.564 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-12-05T22:20:09.565 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00038211 s, 1.3 MB/s 2023-12-05T22:20:09.566 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-05T22:20:09.615 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2023-12-05T22:20:09.661 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-05T22:20:09.662 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:20:09.662 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 915 Links: 1 2023-12-05T22:20:09.662 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:20:09.662 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-12-05 22:14:06.220373097 +0000 2023-12-05T22:20:09.662 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-12-05 22:14:05.848376420 +0000 2023-12-05T22:20:09.662 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-12-05 22:14:05.848376420 +0000 2023-12-05T22:20:09.662 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-12-05T22:20:09.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-05T22:20:09.717 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-12-05T22:20:09.717 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-12-05T22:20:09.717 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00042257 s, 1.2 MB/s 2023-12-05T22:20:09.718 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-05T22:20:09.767 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2023-12-05T22:20:09.814 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-05T22:20:09.814 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-05T22:20:09.814 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 931 Links: 1 2023-12-05T22:20:09.814 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-05T22:20:09.815 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-12-05 22:14:06.784368059 +0000 2023-12-05T22:20:09.815 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-12-05 22:14:06.440371132 +0000 2023-12-05T22:20:09.815 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-12-05 22:14:06.440371132 +0000 2023-12-05T22:20:09.815 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-12-05T22:20:09.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-05T22:20:09.872 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-12-05T22:20:09.872 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-12-05T22:20:09.872 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000300173 s, 1.7 MB/s 2023-12-05T22:20:09.873 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-05T22:20:09.924 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:20:09.924 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-05T22:20:09.925 INFO:tasks.ceph:Generating config... 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client] log to stderr = False 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:09.926 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:09.927 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:09.927 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:09.927 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:09.927 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:09.927 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:09.927 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-05T22:20:09.927 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-05T22:20:09.927 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-05T22:20:09.927 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-05T22:20:09.927 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-05T22:20:09.927 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-05T22:20:09.927 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-05T22:20:09.927 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-05T22:20:09.927 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-05T22:20:09.927 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-05T22:20:09.927 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-12-05T22:20:09.928 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-05T22:20:09.928 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-05T22:20:09.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-05T22:20:10.000 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.keyring 2023-12-05T22:20:10.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-05T22:20:10.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-05T22:20:10.133 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.190')] 2023-12-05T22:20:10.133 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.190', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-05T22:20:10.135 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:10.135 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-05T22:20:10.182 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.190 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:10.261 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:10.261 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: generated fsid 778655ca-dd08-4388-ac14-e9d094947e15 2023-12-05T22:20:10.261 INFO:teuthology.orchestra.run.smithi190.stdout:setting min_mon_release = pacific 2023-12-05T22:20:10.261 INFO:teuthology.orchestra.run.smithi190.stdout:epoch 0 2023-12-05T22:20:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:fsid 778655ca-dd08-4388-ac14-e9d094947e15 2023-12-05T22:20:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:last_changed 2023-12-05T22:20:10.265594+0000 2023-12-05T22:20:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:created 2023-12-05T22:20:10.265594+0000 2023-12-05T22:20:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:min_mon_release 16 (pacific) 2023-12-05T22:20:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:election_strategy: 1 2023-12-05T22:20:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:0: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.a 2023-12-05T22:20:10.262 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-05T22:20:10.263 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-05T22:20:10.309 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 778655ca-dd08-4388-ac14-e9d094947e15... 2023-12-05T22:20:10.310 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-05T22:20:10.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.356 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2023-12-05T22:20:10.357 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2023-12-05T22:20:10.358 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = 172.21.15.190 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: mon client directed command retry = 5 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on pool no app = False 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject socket failures = 5000 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 778655ca-dd08-4388-ac14-e9d094947e15 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2023-12-05T22:20:10.359 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug trim objects = true 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2023-12-05T22:20:10.362 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.362 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2023-12-05T22:20:10.362 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.362 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2023-12-05T22:20:10.362 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2023-12-05T22:20:10.362 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.362 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: osd memory target = 1610612736 2023-12-05T22:20:10.363 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-12-05T22:20:10.364 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-12-05T22:20:10.365 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2023-12-05T22:20:10.365 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-12-05T22:20:10.365 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2023-12-05T22:20:10.365 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2023-12-05T22:20:10.365 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2023-12-05T22:20:10.366 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.367 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2023-12-05T22:20:10.367 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-05T22:20:10.367 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-05T22:20:10.367 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.367 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd = 20 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: debug rbd_mirror = 15 2023-12-05T22:20:10.368 INFO:teuthology.orchestra.run.smithi062.stdout: log to stderr = False 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: rbd default features = 125 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: rbd mirror image policy type = simple 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.0] 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.1] 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.2] 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout:[client.mirror.3] 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.369 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.383 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2023-12-05T22:20:10.383 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2023-12-05T22:20:10.383 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-05T22:20:10.383 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2023-12-05T22:20:10.383 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.383 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2023-12-05T22:20:10.383 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.384 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.385 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = 172.21.15.190 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 5000 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 778655ca-dd08-4388-ac14-e9d094947e15 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.386 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2023-12-05T22:20:10.387 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.388 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: bdev async discard = True 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: bdev enable discard = True 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 1/20 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 1/20 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 4/10 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2023-12-05T22:20:10.389 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2023-12-05T22:20:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2023-12-05T22:20:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2023-12-05T22:20:10.390 INFO:teuthology.orchestra.run.smithi190.stdout: osd memory target = 1610612736 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:20:10.391 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-12-05T22:20:10.392 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2023-12-05T22:20:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2023-12-05T22:20:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-12-05T22:20:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2023-12-05T22:20:10.393 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2023-12-05T22:20:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-12-05T22:20:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-05T22:20:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2023-12-05T22:20:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2023-12-05T22:20:10.394 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.395 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-05T22:20:10.396 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd = 20 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd_mirror = 15 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: log to stderr = False 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: rbd default features = 125 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: rbd mirror image policy type = simple 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.397 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.398 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2023-12-05T22:20:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-05T22:20:10.398 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-05T22:20:10.399 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-05T22:20:10.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-05T22:20:10.489 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-05T22:20:10.489 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:10.490 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-05T22:20:10.538 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:10.538 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-05T22:20:10.586 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T22:20:10.586 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:20:10.587 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-05T22:20:10.587 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-05T22:20:10.611 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:20:10.611 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:10.661 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T22:20:10.661 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:10.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-05T22:20:10.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-05T22:20:10.685 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:10.685 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:10.734 INFO:tasks.ceph:Setting up mon nodes... 2023-12-05T22:20:10.734 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-05T22:20:10.735 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-05T22:20:10.780 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-05T22:20:10.783 INFO:tasks.ceph:Setting up mds nodes... 2023-12-05T22:20:10.784 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-05T22:20:10.784 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-05T22:20:10.860 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-05T22:20:10.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-05T22:20:10.909 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-05T22:20:10.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-05T22:20:10.997 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-05T22:20:11.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-05T22:20:11.079 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-05T22:20:11.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-05T22:20:11.170 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-05T22:20:11.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-05T22:20:11.257 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-05T22:20:11.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-05T22:20:11.338 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-05T22:20:11.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-05T22:20:11.422 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-05T22:20:11.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-05T22:20:11.502 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-05T22:20:11.513 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-05T22:20:11.513 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-05T22:20:11.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-05T22:20:11.571 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:20:11.571 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-05T22:20:11.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T22:20:11.572 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-05T22:20:11.633 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-05T22:20:11.633 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-05T22:20:11.633 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-05T22:20:11.633 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-05T22:20:11.633 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-05T22:20:11.634 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-12-05T22:20:11.634 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-05T22:20:11.634 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-05T22:20:11.634 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-05T22:20:11.634 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-05T22:20:12.092 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-12-05T22:20:12.099 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-12-05T22:20:12.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-05T22:20:12.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-05T22:20:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:20:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2023-12-05T22:20:12.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-05T22:20:12.267 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:20:12.267 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-05T22:20:12.267 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T22:20:12.268 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-05T22:20:12.330 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-05T22:20:12.330 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-05T22:20:12.331 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-05T22:20:12.799 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-12-05T22:20:12.801 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-12-05T22:20:12.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-05T22:20:12.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-05T22:20:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:20:12.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2023-12-05T22:20:12.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-05T22:20:13.034 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-05T22:20:13.034 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-05T22:20:13.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T22:20:13.034 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-05T22:20:13.097 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-05T22:20:13.098 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-05T22:20:13.588 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-12-05T22:20:13.589 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-12-05T22:20:13.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-05T22:20:13.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-05T22:20:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:20:13.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2023-12-05T22:20:13.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:13.857 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:13.861+0000 7ff87eba8640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-05T22:20:13.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:13.861+0000 7ff87eba8640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-05T22:20:13.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:13.861+0000 7ff87eba8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-05T22:20:13.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:13.861+0000 7ff87eba8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-05T22:20:13.858 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:13.861+0000 7ff87eba8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-05T22:20:13.859 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:13.861+0000 7ff87eba8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-05T22:20:16.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-05T22:20:16.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:16.973 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:16.977+0000 7f4148273640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-05T22:20:16.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:16.977+0000 7f4148273640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-05T22:20:16.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:16.977+0000 7f4148273640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-05T22:20:16.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:16.977+0000 7f4148273640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-05T22:20:16.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:16.977+0000 7f4148273640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-05T22:20:16.974 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:16.977+0000 7f4148273640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-05T22:20:19.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-05T22:20:20.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:20.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:20.125+0000 7f88c1b24640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-05T22:20:20.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:20.125+0000 7f88c1b24640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-05T22:20:20.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:20.125+0000 7f88c1b24640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-05T22:20:20.123 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:20.125+0000 7f88c1b24640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-05T22:20:20.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:20.125+0000 7f88c1b24640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-05T22:20:20.124 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T22:20:20.125+0000 7f88c1b24640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-05T22:20:23.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-05T22:20:23.201 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-05T22:20:23.202 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:20:23.202 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-05T22:20:23.217 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-05T22:20:23.256 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-05T22:20:23.307 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-05T22:20:23.359 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-12-05T22:20:23.359 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-05T22:20:23.365 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.365 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-05T22:20:23.405 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.406 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-05T22:20:23.454 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.454 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-05T22:20:23.501 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.502 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-05T22:20:23.550 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.550 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-05T22:20:23.597 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.598 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-05T22:20:23.646 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.646 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-05T22:20:23.694 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:23.694 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-05T22:20:23.741 INFO:tasks.ceph:Adding keys to all mons... 2023-12-05T22:20:23.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-05T22:20:23.796 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2023-12-05T22:20:23.796 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAaom9lNcGtLhAAl2i2S5H1jvmLwciFPp86rA== 2023-12-05T22:20:23.796 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2023-12-05T22:20:23.796 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAdom9lPc9fMxAAJ4PPEOYNxSEkYQRN6O6yxw== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAgom9lpONJOhAAWZ7ckV+DkdgM8Hqi02QZ+g== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAkom9lI6CXBxAAJiz0x0nOej3ek5Bhtp46aQ== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAaom9l/nRmMxAABVtKJgG2Nq1FupnHEvdjFA== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror] 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAaom9lRyh8NhAALMdHFaNWN2VdTnDw/VurLQ== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAaom9lsJiQOxAAIiuZO1NXkyG17YbHfZHsLQ== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9lnM8CBRAApHcHNydj0XZ60HKyU1/QeQ== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9l4nlpChAAZuMW7Xec1OGBDOk3GYU8lw== 2023-12-05T22:20:23.797 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2023-12-05T22:20:23.798 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9llWWXDxAAJL+8qCGlIVMwOHxoDCrQkg== 2023-12-05T22:20:23.798 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.4] 2023-12-05T22:20:23.798 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9lVqZwFBAA5S420IgEhzHo/lxl/vYLxw== 2023-12-05T22:20:23.798 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.5] 2023-12-05T22:20:23.798 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9lyXByGRAAeA0zRY0ssfbITAMkEM9Gyg== 2023-12-05T22:20:23.798 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.6] 2023-12-05T22:20:23.798 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9lz0UuHhAARy7ZL+qx6voJ6AcKCgKV0g== 2023-12-05T22:20:23.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-05T22:20:23.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-05T22:20:23.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-05T22:20:24.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-05T22:20:24.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-05T22:20:24.778 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-05T22:20:24.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-05T22:20:24.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-05T22:20:24.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-05T22:20:24.939 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-05T22:20:24.989 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-05T22:20:24.989 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-05T22:20:24.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-05T22:20:25.034 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-05T22:20:25.035 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-05T22:20:25.035 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-05T22:20:25.035 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-05T22:20:25.038 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-05T22:20:25.039 INFO:tasks.ceph:Setting crush tunables to default 2023-12-05T22:20:25.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-05T22:20:25.287 INFO:teuthology.orchestra.run.smithi190.stderr:adjusted tunables profile to default 2023-12-05T22:20:25.307 INFO:tasks.ceph:check_enable_crimson: False 2023-12-05T22:20:25.307 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-05T22:20:25.307 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:25.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-05T22:20:25.319 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:25.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-05T22:20:25.371 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:20:25.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-05T22:20:25.408 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:25.407+0000 7fb1b5ad6180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-05T22:20:25.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd new 692dc9ad-0ab0-4154-a5dc-e44042591dba 0 2023-12-05T22:20:25.672 INFO:teuthology.orchestra.run.smithi190.stdout:0 2023-12-05T22:20:25.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd new 17d4c62d-069d-4e4a-9160-fb7a1e1c7848 1 2023-12-05T22:20:25.800 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:25.799+0000 7fb1b5ad6180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-05T22:20:25.883 INFO:teuthology.orchestra.run.smithi190.stdout:1 2023-12-05T22:20:25.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd new 96bc702c-c283-42cc-b2c4-ad072b2ea8b9 2 2023-12-05T22:20:25.935 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:25.931+0000 7fb1b5ad6180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-05T22:20:26.107 INFO:teuthology.orchestra.run.smithi190.stdout:2 2023-12-05T22:20:26.126 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-05T22:20:26.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-05T22:20:26.128 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-05T22:20:26.128 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-05T22:20:26.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-05T22:20:26.131 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-05T22:20:26.131 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-05T22:20:26.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-05T22:20:26.134 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-05T22:20:26.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-05T22:20:26.319 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:26.319 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":5,"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","created":"2023-12-05T22:20:25.182935+0000","modified":"2023-12-05T22:20:26.110836+0000","last_up_change":"0.000000","last_in_change":"2023-12-05T22:20:26.110836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"692dc9ad-0ab0-4154-a5dc-e44042591dba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17d4c62d-069d-4e4a-9160-fb7a1e1c7848","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96bc702c-c283-42cc-b2c4-ad072b2ea8b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:26.336 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-05T22:20:26.336 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-05T22:20:26.375 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.371+0000 7fb1b5ad6180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-05T22:20:26.427 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.427+0000 7fb1b5ad6180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-05T22:20:26.478 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.475+0000 7fb1b5ad6180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-05T22:20:26.555 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.555+0000 7fb1b5ad6180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-05T22:20:26.607 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.603+0000 7fb1b5ad6180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-05T22:20:26.638 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-05T22:20:26.682 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.679+0000 7fb1b5ad6180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-05T22:20:26.733 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.731+0000 7fb1b5ad6180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-05T22:20:26.738 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:20:26.741+0000 7fdd686ed640 -1 Falling back to public interface 2023-12-05T22:20:26.739 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T22:20:26.741+0000 7fa64f55a640 -1 Falling back to public interface 2023-12-05T22:20:26.739 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:20:26.741+0000 7fe90112c640 -1 Falling back to public interface 2023-12-05T22:20:26.739 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:20:26.741+0000 7fe9000ec640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2023-12-05T22:20:26.739 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:20:26.741+0000 7fdd676ad640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6810/0: (98) Address already in use 2023-12-05T22:20:26.824 INFO:teuthology.misc.health.smithi190.stdout: 2023-12-05T22:20:26.825 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":5,"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","created":"2023-12-05T22:20:25.182935+0000","modified":"2023-12-05T22:20:26.110836+0000","last_up_change":"0.000000","last_in_change":"2023-12-05T22:20:26.110836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"692dc9ad-0ab0-4154-a5dc-e44042591dba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17d4c62d-069d-4e4a-9160-fb7a1e1c7848","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96bc702c-c283-42cc-b2c4-ad072b2ea8b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:26.840 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-05T22:20:26.867 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.863+0000 7fb1b5ad6180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-05T22:20:26.920 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:26.919+0000 7fb1b5ad6180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-05T22:20:27.143 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-05T22:20:27.143 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-05T22:20:27.143 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-12-05T22:20:27.151 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:27.147+0000 7fb1b5ad6180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-05T22:20:27.278 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:27.275+0000 7fb1b5ad6180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-05T22:20:27.391 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:27.391+0000 7fb1b5ad6180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-05T22:20:27.455 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:27.451+0000 7fb1b5ad6180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-05T22:20:27.582 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:27.579+0000 7fb1b5ad6180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-05T22:20:28.075 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:28.071+0000 7fb1b5ad6180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-05T22:20:28.151 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:28.147+0000 7fb1b5ad6180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-05T22:20:28.290 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:28.287+0000 7fb1b5ad6180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-05T22:20:28.365 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:28.363+0000 7fb1b5ad6180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-05T22:20:28.654 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:28.651+0000 7fb1b5ad6180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-05T22:20:28.709 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:28.707+0000 7fb1b5ad6180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-05T22:20:29.725 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T22:20:29.723+0000 7fb15d249640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-05T22:20:30.569 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:20:30.573+0000 7fdd686ed640 -1 osd.0 0 log_to_monitors true 2023-12-05T22:20:30.573 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T22:20:30.577+0000 7fa64f55a640 -1 osd.1 0 log_to_monitors true 2023-12-05T22:20:30.588 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:20:30.589+0000 7fe90112c640 -1 osd.2 0 log_to_monitors true 2023-12-05T22:20:33.144 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-05T22:20:33.578 INFO:teuthology.misc.health.smithi190.stdout: 2023-12-05T22:20:33.578 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":7,"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","created":"2023-12-05T22:20:25.182935+0000","modified":"2023-12-05T22:20:32.139756+0000","last_up_change":"0.000000","last_in_change":"2023-12-05T22:20:26.110836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"692dc9ad-0ab0-4154-a5dc-e44042591dba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17d4c62d-069d-4e4a-9160-fb7a1e1c7848","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96bc702c-c283-42cc-b2c4-ad072b2ea8b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:33.603 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-05T22:20:34.120 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T22:20:34.120+0000 7fa64acde640 -1 osd.1 0 waiting for initial osdmap 2023-12-05T22:20:34.123 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:20:34.124+0000 7fe8fe0e8640 -1 osd.2 0 waiting for initial osdmap 2023-12-05T22:20:34.141 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T22:20:34.144+0000 7fa643325640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-05T22:20:34.157 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:20:34.160+0000 7fe8f4ef7640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-05T22:20:34.168 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:20:34.168+0000 7fdd63e71640 -1 osd.0 0 waiting for initial osdmap 2023-12-05T22:20:34.206 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:20:34.208+0000 7fdd5c4b8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-05T22:20:39.906 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-05T22:20:40.198 INFO:teuthology.misc.health.smithi190.stdout: 2023-12-05T22:20:40.198 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":12,"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","created":"2023-12-05T22:20:25.182935+0000","modified":"2023-12-05T22:20:39.207111+0000","last_up_change":"2023-12-05T22:20:35.134461+0000","last_in_change":"2023-12-05T22:20:26.110836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-05T22:20:36.801123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"692dc9ad-0ab0-4154-a5dc-e44042591dba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6802","nonce":932525248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6806","nonce":932525248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6815","nonce":932525248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6811","nonce":932525248}]},"public_addr":"172.21.15.190:6802/932525248","cluster_addr":"172.21.15.190:6806/932525248","heartbeat_back_addr":"172.21.15.190:6815/932525248","heartbeat_front_addr":"172.21.15.190:6811/932525248","state":["exists","up"]},{"osd":1,"uuid":"17d4c62d-069d-4e4a-9160-fb7a1e1c7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6804","nonce":2936208001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6810","nonce":2936208001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6819","nonce":2936208001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6816","nonce":2936208001}]},"public_addr":"172.21.15.190:6804/2936208001","cluster_addr":"172.21.15.190:6810/2936208001","heartbeat_back_addr":"172.21.15.190:6819/2936208001","heartbeat_front_addr":"172.21.15.190:6816/2936208001","state":["exists","up"]},{"osd":2,"uuid":"96bc702c-c283-42cc-b2c4-ad072b2ea8b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6809","nonce":1085524115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6817","nonce":1085524115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6823","nonce":1085524115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6821","nonce":1085524115}]},"public_addr":"172.21.15.190:6809/1085524115","cluster_addr":"172.21.15.190:6817/1085524115","heartbeat_back_addr":"172.21.15.190:6823/1085524115","heartbeat_front_addr":"172.21.15.190:6821/1085524115","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.623720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.626219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.574104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:40.216 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-05T22:20:40.216 INFO:tasks.ceph:Creating RBD pool 2023-12-05T22:20:40.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-05T22:20:41.221 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'rbd' created 2023-12-05T22:20:41.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-05T22:20:42.231 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-05T22:20:42.259 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-05T22:20:42.259 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-05T22:20:42.259 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-05T22:20:42.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-05T22:20:42.260 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-05T22:20:42.590 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:42.608 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":223342419},{"type":"v1","addr":"172.21.15.62:6827","nonce":223342419}]},"active_addr":"172.21.15.62:6827/223342419","active_change":"2023-12-05T22:20:28.715964+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":3862136127}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":117275722}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2635479878}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2177671399}]}]} 2023-12-05T22:20:42.610 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-05T22:20:42.611 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-05T22:20:42.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-05T22:20:42.904 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:42.905 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":14,"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","created":"2023-12-05T22:20:25.182935+0000","modified":"2023-12-05T22:20:42.226595+0000","last_up_change":"2023-12-05T22:20:35.134461+0000","last_in_change":"2023-12-05T22:20:26.110836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-05T22:20:36.801123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-05T22:20:40.543475+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"692dc9ad-0ab0-4154-a5dc-e44042591dba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6802","nonce":932525248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6806","nonce":932525248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6815","nonce":932525248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6811","nonce":932525248}]},"public_addr":"172.21.15.190:6802/932525248","cluster_addr":"172.21.15.190:6806/932525248","heartbeat_back_addr":"172.21.15.190:6815/932525248","heartbeat_front_addr":"172.21.15.190:6811/932525248","state":["exists","up"]},{"osd":1,"uuid":"17d4c62d-069d-4e4a-9160-fb7a1e1c7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6804","nonce":2936208001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6810","nonce":2936208001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6819","nonce":2936208001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6816","nonce":2936208001}]},"public_addr":"172.21.15.190:6804/2936208001","cluster_addr":"172.21.15.190:6810/2936208001","heartbeat_back_addr":"172.21.15.190:6819/2936208001","heartbeat_front_addr":"172.21.15.190:6816/2936208001","state":["exists","up"]},{"osd":2,"uuid":"96bc702c-c283-42cc-b2c4-ad072b2ea8b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6809","nonce":1085524115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6817","nonce":1085524115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6823","nonce":1085524115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6821","nonce":1085524115}]},"public_addr":"172.21.15.190:6809/1085524115","cluster_addr":"172.21.15.190:6817/1085524115","heartbeat_back_addr":"172.21.15.190:6823/1085524115","heartbeat_front_addr":"172.21.15.190:6821/1085524115","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.623720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.626219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.574104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:42.923 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-05T22:20:42.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-05T22:20:43.198 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:43.199 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":14,"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","created":"2023-12-05T22:20:25.182935+0000","modified":"2023-12-05T22:20:42.226595+0000","last_up_change":"2023-12-05T22:20:35.134461+0000","last_in_change":"2023-12-05T22:20:26.110836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-05T22:20:36.801123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-05T22:20:40.543475+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"692dc9ad-0ab0-4154-a5dc-e44042591dba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6802","nonce":932525248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6806","nonce":932525248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6815","nonce":932525248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":932525248},{"type":"v1","addr":"172.21.15.190:6811","nonce":932525248}]},"public_addr":"172.21.15.190:6802/932525248","cluster_addr":"172.21.15.190:6806/932525248","heartbeat_back_addr":"172.21.15.190:6815/932525248","heartbeat_front_addr":"172.21.15.190:6811/932525248","state":["exists","up"]},{"osd":1,"uuid":"17d4c62d-069d-4e4a-9160-fb7a1e1c7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6804","nonce":2936208001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6810","nonce":2936208001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6819","nonce":2936208001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2936208001},{"type":"v1","addr":"172.21.15.190:6816","nonce":2936208001}]},"public_addr":"172.21.15.190:6804/2936208001","cluster_addr":"172.21.15.190:6810/2936208001","heartbeat_back_addr":"172.21.15.190:6819/2936208001","heartbeat_front_addr":"172.21.15.190:6816/2936208001","state":["exists","up"]},{"osd":2,"uuid":"96bc702c-c283-42cc-b2c4-ad072b2ea8b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6809","nonce":1085524115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6817","nonce":1085524115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6823","nonce":1085524115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1085524115},{"type":"v1","addr":"172.21.15.190:6821","nonce":1085524115}]},"public_addr":"172.21.15.190:6809/1085524115","cluster_addr":"172.21.15.190:6817/1085524115","heartbeat_back_addr":"172.21.15.190:6823/1085524115","heartbeat_front_addr":"172.21.15.190:6821/1085524115","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.623720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.626219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:20:31.574104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T22:20:43.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-05T22:20:43.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-05T22:20:43.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-05T22:20:43.364 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2023-12-05T22:20:43.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-05T22:20:43.376 INFO:teuthology.orchestra.run.smithi190.stdout:34359738374 2023-12-05T22:20:43.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-05T22:20:43.382 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2023-12-05T22:20:43.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-05T22:20:43.699 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2023-12-05T22:20:43.717 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2023-12-05T22:20:43.722 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2023-12-05T22:20:43.734 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2023-12-05T22:20:43.781 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2023-12-05T22:20:43.798 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-05T22:20:44.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-05T22:20:44.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-05T22:20:44.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-05T22:20:45.024 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2023-12-05T22:20:45.041 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2023-12-05T22:20:45.055 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2023-12-05T22:20:45.072 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2023-12-05T22:20:45.104 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2023-12-05T22:20:45.121 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-05T22:20:46.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-05T22:20:46.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-05T22:20:46.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-05T22:20:46.361 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2023-12-05T22:20:46.378 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.1 2023-12-05T22:20:46.379 DEBUG:teuthology.parallel:result is None 2023-12-05T22:20:46.397 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2023-12-05T22:20:46.413 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2023-12-05T22:20:46.413 DEBUG:teuthology.parallel:result is None 2023-12-05T22:20:46.431 INFO:teuthology.orchestra.run.smithi190.stdout:34359738374 2023-12-05T22:20:46.449 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2023-12-05T22:20:46.449 DEBUG:teuthology.parallel:result is None 2023-12-05T22:20:46.449 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-05T22:20:46.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T22:20:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:46.762 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T22:20:46.778 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-05T22:20:46.726182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82608,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032912,"statfs":{"total":289910292480,"available":289825701888,"internally_reserved":0,"allocated":1581056,"data_stored":1292366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001508"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.383109+0000","last_change":"2023-12-05T22:20:42.250265+0000","last_active":"2023-12-05T22:20:43.383109+0000","last_peered":"2023-12-05T22:20:43.383109+0000","last_clean":"2023-12-05T22:20:43.383109+0000","last_became_active":"2023-12-05T22:20:42.249844+0000","last_became_peered":"2023-12-05T22:20:42.249844+0000","last_unstale":"2023-12-05T22:20:43.383109+0000","last_undegraded":"2023-12-05T22:20:43.383109+0000","last_fullsized":"2023-12-05T22:20:43.383109+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T04:19:24.693215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382982+0000","last_change":"2023-12-05T22:20:42.249082+0000","last_active":"2023-12-05T22:20:43.382982+0000","last_peered":"2023-12-05T22:20:43.382982+0000","last_clean":"2023-12-05T22:20:43.382982+0000","last_became_active":"2023-12-05T22:20:42.248652+0000","last_became_peered":"2023-12-05T22:20:42.248652+0000","last_unstale":"2023-12-05T22:20:43.382982+0000","last_undegraded":"2023-12-05T22:20:43.382982+0000","last_fullsized":"2023-12-05T22:20:43.382982+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:11:07.396035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382979+0000","last_change":"2023-12-05T22:20:42.248949+0000","last_active":"2023-12-05T22:20:43.382979+0000","last_peered":"2023-12-05T22:20:43.382979+0000","last_clean":"2023-12-05T22:20:43.382979+0000","last_became_active":"2023-12-05T22:20:42.248499+0000","last_became_peered":"2023-12-05T22:20:42.248499+0000","last_unstale":"2023-12-05T22:20:43.382979+0000","last_undegraded":"2023-12-05T22:20:43.382979+0000","last_fullsized":"2023-12-05T22:20:43.382979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:26:35.448173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382938+0000","last_change":"2023-12-05T22:20:42.250219+0000","last_active":"2023-12-05T22:20:43.382938+0000","last_peered":"2023-12-05T22:20:43.382938+0000","last_clean":"2023-12-05T22:20:43.382938+0000","last_became_active":"2023-12-05T22:20:42.249717+0000","last_became_peered":"2023-12-05T22:20:42.249717+0000","last_unstale":"2023-12-05T22:20:43.382938+0000","last_undegraded":"2023-12-05T22:20:43.382938+0000","last_fullsized":"2023-12-05T22:20:43.382938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T04:47:33.739798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.246350+0000","last_change":"2023-12-05T22:20:42.246835+0000","last_active":"2023-12-05T22:20:43.246350+0000","last_peered":"2023-12-05T22:20:43.246350+0000","last_clean":"2023-12-05T22:20:43.246350+0000","last_became_active":"2023-12-05T22:20:42.246323+0000","last_became_peered":"2023-12-05T22:20:42.246323+0000","last_unstale":"2023-12-05T22:20:43.246350+0000","last_undegraded":"2023-12-05T22:20:43.246350+0000","last_fullsized":"2023-12-05T22:20:43.246350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:59.353905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.478124+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T22:20:43.478124+0000","last_peered":"2023-12-05T22:20:43.478124+0000","last_clean":"2023-12-05T22:20:43.478124+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T22:20:43.478124+0000","last_undegraded":"2023-12-05T22:20:43.478124+0000","last_fullsized":"2023-12-05T22:20:43.478124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.478124+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T22:20:43.478124+0000","last_peered":"2023-12-05T22:20:43.478124+0000","last_clean":"2023-12-05T22:20:43.478124+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T22:20:43.478124+0000","last_undegraded":"2023-12-05T22:20:43.478124+0000","last_fullsized":"2023-12-05T22:20:43.478124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382985+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T22:20:43.382985+0000","last_peered":"2023-12-05T22:20:43.382985+0000","last_clean":"2023-12-05T22:20:43.382985+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T22:20:43.382985+0000","last_undegraded":"2023-12-05T22:20:43.382985+0000","last_fullsized":"2023-12-05T22:20:43.382985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:42.245287+0000","last_change":"2023-12-05T22:20:38.479084+0000","last_active":"2023-12-05T22:20:42.245287+0000","last_peered":"2023-12-05T22:20:42.245287+0000","last_clean":"2023-12-05T22:20:42.245287+0000","last_became_active":"2023-12-05T22:20:38.478390+0000","last_became_peered":"2023-12-05T22:20:38.478390+0000","last_unstale":"2023-12-05T22:20:42.245287+0000","last_undegraded":"2023-12-05T22:20:42.245287+0000","last_fullsized":"2023-12-05T22:20:42.245287+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T04:21:21.272496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T22:20:46.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T22:20:47.056 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:47.056 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T22:20:47.072 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-05T22:20:46.726182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82608,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032912,"statfs":{"total":289910292480,"available":289825701888,"internally_reserved":0,"allocated":1581056,"data_stored":1292366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001508"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.383109+0000","last_change":"2023-12-05T22:20:42.250265+0000","last_active":"2023-12-05T22:20:43.383109+0000","last_peered":"2023-12-05T22:20:43.383109+0000","last_clean":"2023-12-05T22:20:43.383109+0000","last_became_active":"2023-12-05T22:20:42.249844+0000","last_became_peered":"2023-12-05T22:20:42.249844+0000","last_unstale":"2023-12-05T22:20:43.383109+0000","last_undegraded":"2023-12-05T22:20:43.383109+0000","last_fullsized":"2023-12-05T22:20:43.383109+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T04:19:24.693215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382982+0000","last_change":"2023-12-05T22:20:42.249082+0000","last_active":"2023-12-05T22:20:43.382982+0000","last_peered":"2023-12-05T22:20:43.382982+0000","last_clean":"2023-12-05T22:20:43.382982+0000","last_became_active":"2023-12-05T22:20:42.248652+0000","last_became_peered":"2023-12-05T22:20:42.248652+0000","last_unstale":"2023-12-05T22:20:43.382982+0000","last_undegraded":"2023-12-05T22:20:43.382982+0000","last_fullsized":"2023-12-05T22:20:43.382982+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:11:07.396035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382979+0000","last_change":"2023-12-05T22:20:42.248949+0000","last_active":"2023-12-05T22:20:43.382979+0000","last_peered":"2023-12-05T22:20:43.382979+0000","last_clean":"2023-12-05T22:20:43.382979+0000","last_became_active":"2023-12-05T22:20:42.248499+0000","last_became_peered":"2023-12-05T22:20:42.248499+0000","last_unstale":"2023-12-05T22:20:43.382979+0000","last_undegraded":"2023-12-05T22:20:43.382979+0000","last_fullsized":"2023-12-05T22:20:43.382979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:26:35.448173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382938+0000","last_change":"2023-12-05T22:20:42.250219+0000","last_active":"2023-12-05T22:20:43.382938+0000","last_peered":"2023-12-05T22:20:43.382938+0000","last_clean":"2023-12-05T22:20:43.382938+0000","last_became_active":"2023-12-05T22:20:42.249717+0000","last_became_peered":"2023-12-05T22:20:42.249717+0000","last_unstale":"2023-12-05T22:20:43.382938+0000","last_undegraded":"2023-12-05T22:20:43.382938+0000","last_fullsized":"2023-12-05T22:20:43.382938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T04:47:33.739798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.246350+0000","last_change":"2023-12-05T22:20:42.246835+0000","last_active":"2023-12-05T22:20:43.246350+0000","last_peered":"2023-12-05T22:20:43.246350+0000","last_clean":"2023-12-05T22:20:43.246350+0000","last_became_active":"2023-12-05T22:20:42.246323+0000","last_became_peered":"2023-12-05T22:20:42.246323+0000","last_unstale":"2023-12-05T22:20:43.246350+0000","last_undegraded":"2023-12-05T22:20:43.246350+0000","last_fullsized":"2023-12-05T22:20:43.246350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:01:59.353905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.478124+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T22:20:43.478124+0000","last_peered":"2023-12-05T22:20:43.478124+0000","last_clean":"2023-12-05T22:20:43.478124+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T22:20:43.478124+0000","last_undegraded":"2023-12-05T22:20:43.478124+0000","last_fullsized":"2023-12-05T22:20:43.478124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.478124+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T22:20:43.478124+0000","last_peered":"2023-12-05T22:20:43.478124+0000","last_clean":"2023-12-05T22:20:43.478124+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T22:20:43.478124+0000","last_undegraded":"2023-12-05T22:20:43.478124+0000","last_fullsized":"2023-12-05T22:20:43.478124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-05T22:20:43.382985+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T22:20:43.382985+0000","last_peered":"2023-12-05T22:20:43.382985+0000","last_clean":"2023-12-05T22:20:43.382985+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T22:20:43.382985+0000","last_undegraded":"2023-12-05T22:20:43.382985+0000","last_fullsized":"2023-12-05T22:20:43.382985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-05T22:20:42.245287+0000","last_change":"2023-12-05T22:20:38.479084+0000","last_active":"2023-12-05T22:20:42.245287+0000","last_peered":"2023-12-05T22:20:42.245287+0000","last_clean":"2023-12-05T22:20:42.245287+0000","last_became_active":"2023-12-05T22:20:38.478390+0000","last_became_peered":"2023-12-05T22:20:38.478390+0000","last_unstale":"2023-12-05T22:20:42.245287+0000","last_undegraded":"2023-12-05T22:20:42.245287+0000","last_fullsized":"2023-12-05T22:20:42.245287+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T04:21:21.272496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T22:20:47.073 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-05T22:20:47.073 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-05T22:20:47.074 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-05T22:20:47.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-05T22:20:47.398 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T22:20:47.399 INFO:teuthology.orchestra.run.smithi190.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-05T22:20:47.417 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-05T22:20:47.418 INFO:teuthology.run_tasks:Running task exec... 2023-12-05T22:20:47.429 INFO:teuthology.task.exec:Executing custom commands... 2023-12-05T22:20:47.429 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T22:20:47.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:47.776 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror] 2023-12-05T22:20:47.777 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQD0oW9lJCsZKhAAxLm9g9Qh1U0sEwopzaoReQ== 2023-12-05T22:20:47.777 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-05T22:20:47.777 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:47.777 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror 2023-12-05T22:20:47.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:48.137 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2023-12-05T22:20:48.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQD0oW9lj2+eMBAA4DRCV3jej7AlbiyvnT9pLQ== 2023-12-05T22:20:48.137 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:48.137 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:48.138 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.0 2023-12-05T22:20:48.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:48.498 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2023-12-05T22:20:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQD0oW9lLN2+NRAARJkTn4N4FjagvgrVbgJiRw== 2023-12-05T22:20:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:48.499 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:48.499 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.1 2023-12-05T22:20:48.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:48.878 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2023-12-05T22:20:48.878 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQD0oW9l0Qe7OhAAO8Uwj35ORjG7jGz8G8gqbg== 2023-12-05T22:20:48.878 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:48.878 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:48.878 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.2 2023-12-05T22:20:48.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:49.238 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2023-12-05T22:20:49.239 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQD1oW9l78tWBBAAHQRjRh4X+jPT166GyGPHHw== 2023-12-05T22:20:49.239 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:49.239 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:49.239 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.3 2023-12-05T22:20:49.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:49.590 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror] 2023-12-05T22:20:49.590 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAaom9lRyh8NhAALMdHFaNWN2VdTnDw/VurLQ== 2023-12-05T22:20:49.590 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-05T22:20:49.590 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:49.590 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror 2023-12-05T22:20:49.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:49.934 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2023-12-05T22:20:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAaom9lsJiQOxAAIiuZO1NXkyG17YbHfZHsLQ== 2023-12-05T22:20:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:49.935 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:49.935 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.0 2023-12-05T22:20:49.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:50.301 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2023-12-05T22:20:50.302 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9lnM8CBRAApHcHNydj0XZ60HKyU1/QeQ== 2023-12-05T22:20:50.302 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:50.302 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:50.302 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.1 2023-12-05T22:20:50.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:50.661 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2023-12-05T22:20:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9l4nlpChAAZuMW7Xec1OGBDOk3GYU8lw== 2023-12-05T22:20:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:50.662 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.2 2023-12-05T22:20:50.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-05T22:20:51.027 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2023-12-05T22:20:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAbom9llWWXDxAAJL+8qCGlIVMwOHxoDCrQkg== 2023-12-05T22:20:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "profile rbd-mirror" 2023-12-05T22:20:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "profile rbd" 2023-12-05T22:20:51.027 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.3 2023-12-05T22:20:51.045 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-05T22:20:51.057 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-05T22:20:51.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-05T22:20:51.059 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-05T22:20:51.077 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-05T22:20:51.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-05T22:20:51.079 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-05T22:20:51.088 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-05T22:20:51.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-05T22:20:51.090 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-05T22:20:51.100 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-05T22:20:51.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-05T22:20:51.102 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-05T22:20:51.221 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-05T22:20:51.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-05T22:20:51.224 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:20:51.233 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:20:51.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:20:51.235 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-05T22:20:51.245 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-05T22:20:51.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-05T22:20:51.248 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-05T22:20:51.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-05T22:20:51.257 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-05T22:20:51.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-05T22:20:51.260 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-05T22:20:51.271 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701814851 2023-12-05T22:20:51.272 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-05T22:20:51.272 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-05T22:20:51.282 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701814851 2023-12-05T22:20:51.282 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-05T22:20:51.282 INFO:teuthology.run_tasks:Running task workunit... 2023-12-05T22:20:51.294 INFO:tasks.workunit:Pulling workunits from ref 4a50e96e7d9d85da28ef008617cb8342574bfd1b 2023-12-05T22:20:51.295 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-05T22:20:51.295 DEBUG:teuthology.orchestra.run.smithi190:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-05T22:20:51.295 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-05T22:20:51.296 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-05T22:20:51.296 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-05T22:20:51.296 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-05T22:20:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T22:20:51.301 INFO:teuthology.orchestra.run.smithi190.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-05T22:20:51.301 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-05T22:20:51.348 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-05T22:20:51.348 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-05T22:20:51.395 INFO:tasks.workunit:timeout=6h 2023-12-05T22:20:51.395 INFO:tasks.workunit:cleanup=True 2023-12-05T22:20:51.396 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 4a50e96e7d9d85da28ef008617cb8342574bfd1b 2023-12-05T22:20:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-05T22:21:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Note: switching to '4a50e96e7d9d85da28ef008617cb8342574bfd1b'. 2023-12-05T22:21:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:21:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-05T22:21:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-05T22:21:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:state without impacting any branches by switching back to a branch. 2023-12-05T22:21:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:21:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: git switch -c 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Or undo this operation with: 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: git switch - 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:21:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:HEAD is now at 4a50e96e7d9 Revert "common: hide asio header from common/error_code.h" 2023-12-05T22:21:38.373 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-05T22:21:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-05T22:21:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-05T22:21:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-05T22:21:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-05T22:21:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-05T22:21:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-05T22:21:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-05T22:21:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-05T22:21:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-05T22:21:39.526 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-12-05T22:21:39.526 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-05T22:21:39.572 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-05T22:21:39.573 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-05T22:21:39.574 DEBUG:teuthology.orchestra.run.smithi190:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4a50e96e7d9d85da28ef008617cb8342574bfd1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-05T22:21:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ IMAGE_COUNT=50 2023-12-05T22:21:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ export LOCKDEP=0 2023-12-05T22:21:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-05T22:21:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-05T22:21:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ type xmlstarlet 2023-12-05T22:21:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ XMLSTARLET=xmlstarlet 2023-12-05T22:21:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ RBD_MIRROR_INSTANCES=4 2023-12-05T22:21:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CLUSTER1=cluster1 2023-12-05T22:21:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CLUSTER2=cluster2 2023-12-05T22:21:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ PEER_CLUSTER_SUFFIX= 2023-12-05T22:21:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ POOL=mirror 2023-12-05T22:21:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ PARENT_POOL=mirror_parent 2023-12-05T22:21:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ NS1=ns1 2023-12-05T22:21:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ NS2=ns2 2023-12-05T22:21:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ TEMPDIR= 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ID=mirror 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ MIRROR_USER_ID_PREFIX=mirror. 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ RBD_MIRROR_MODE=snapshot 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ MIRROR_POOL_MODE=image 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ MIRROR_POOL_MODE=image 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ export CEPH_ARGS=--id mirror 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ LAST_MIRROR_INSTANCE=3 2023-12-05T22:21:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-05T22:21:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-05T22:21:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-05T22:21:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_BIN=. 2023-12-05T22:21:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_SRC=. 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -e CMakeCache.txt ] 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -gt 0 ] 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local c 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ trap cleanup $? INT TERM EXIT 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup_tempdir 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mktemp -d 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ TEMPDIR=/tmp/tmp.yK28jjcyo4 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -z 1 ] 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup_pools cluster1 cluster2 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster2 2023-12-05T22:21:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_map_file 2023-12-05T22:21:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_addr 2023-12-05T22:21:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local admin_key_file 2023-12-05T22:21:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid 2023-12-05T22:21:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-05T22:21:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' already exists 2023-12-05T22:21:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-05T22:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' already exists 2023-12-05T22:21:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror 2023-12-05T22:21:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror_parent 2023-12-05T22:21:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-05T22:21:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-05T22:21:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-05T22:21:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-05T22:21:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-05T22:21:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -z ] 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -z ] 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster1 mirror cluster2 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster2 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster2 2023-12-05T22:21:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2023-12-05T22:21:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2023-12-05T22:21:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2023-12-05T22:21:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:21:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-05T22:21:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=8c8a266e-bd47-4f4e-af77-66406330ebb1 2023-12-05T22:21:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T22:21:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:21:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 17 ] 2023-12-05T22:21:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2023-12-05T22:21:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:21:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-05T22:21:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror_parent 2023-12-05T22:21:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster2 2023-12-05T22:21:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster2 2023-12-05T22:21:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2023-12-05T22:21:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2023-12-05T22:21:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2023-12-05T22:21:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:21:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-05T22:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=a92c6d9b-946f-4025-b77d-79fb3a39d0d0 2023-12-05T22:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T22:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 17 ] 2023-12-05T22:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2023-12-05T22:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup_pools cluster2 cluster1 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster1 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_map_file 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_addr 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local admin_key_file 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid 2023-12-05T22:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-05T22:21:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' already exists 2023-12-05T22:21:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-05T22:21:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' already exists 2023-12-05T22:21:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror 2023-12-05T22:21:51.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:21:51.251+0000 7f38012ac880 -1 rbd::mirror::PoolReplayer: 0x55a8c4dbcc00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:51.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:21:51.255+0000 7f3777742880 -1 rbd::mirror::PoolReplayer: 0x56429767cc00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:51.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:21:51.263+0000 7f898003b880 -1 rbd::mirror::PoolReplayer: 0x562f140a4c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:51.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:21:51.383+0000 7f38012ac880 -1 rbd::mirror::PoolReplayer: 0x55a8c5a7d500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:51.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:21:51.431+0000 7f8d2c763880 -1 rbd::mirror::PoolReplayer: 0x561958a16c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:51.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:21:51.535+0000 7f3777742880 -1 rbd::mirror::PoolReplayer: 0x5642977fd500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:51.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:21:51.539+0000 7f898003b880 -1 rbd::mirror::PoolReplayer: 0x562f14d61500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:51.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:21:51.707+0000 7f8d2c763880 -1 rbd::mirror::PoolReplayer: 0x5619596eb500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-05T22:21:53.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.30533524982406535) > weight (0.25) 2023-12-05T22:21:53.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9685101633226687) > weight (0.25) 2023-12-05T22:21:53.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5639396316390287) > weight (0.25) 2023-12-05T22:21:53.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.457781120139374) > weight (0.25) 2023-12-05T22:21:53.297 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-05T22:21:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror_parent 2023-12-05T22:21:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-05T22:21:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-05T22:21:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-05T22:21:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-05T22:21:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-05T22:21:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-05T22:21:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -z ] 2023-12-05T22:21:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -z ] 2023-12-05T22:21:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster2 mirror cluster1 2023-12-05T22:21:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:21:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster1 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster1 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:21:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-05T22:21:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=b215fad4-c0ef-4b17-8e9d-5f3d4dc62e58 2023-12-05T22:21:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T22:21:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 17 ] 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror_parent 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster1 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster1 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2023-12-05T22:21:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2023-12-05T22:21:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:21:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=214b66d9-2a80-47b8-a5d4-0b28618aaae5 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 17 ] 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:21:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirrors cluster1 2023-12-05T22:21:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:21:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:0 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:21:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:1 2023-12-05T22:21:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:21:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:21:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:21:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:2 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:21:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:3 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirrors cluster2 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:21:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:0 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:0 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:0 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster2:0 ] 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:1 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:1 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:1 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster2:1 ] 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:2 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:2 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:2 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster2:2 ] 2023-12-05T22:21:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:3 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:3 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:3 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster2:3 ] 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog TEST: add image and test replay after client crashes 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ date +%F %T 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.yK28jjcyo4/rbd-mirror.test.log 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2023-12-05 22:21:56 TEST: add image and test replay after client crashes 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2023-12-05 22:21:56 TEST: add image and test replay after client crashes 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=test 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror test 512M 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:21:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:21:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2023-12-05T22:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-05T22:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:21:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:21:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:21:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-05T22:21:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:21:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10b993dd6377 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10b993dd6377 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:21:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:21:56 2023 2023-12-05T22:21:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:21:56 2023 2023-12-05T22:21:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:21:56 2023 2023-12-05T22:21:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:21:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:21:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:21:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:21:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:21:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:21:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:21:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ localgrep instance=cluster1 -i 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:21:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:21:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:21:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:21:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:21:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:21:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:21:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:21:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:21:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:21:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:21:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:21:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:21:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:21:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:21:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:21:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:21:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:21:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:21:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:21:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:21:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:21:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:21:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:21:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:21:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:21:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:21:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:21:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:21:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:21:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:21:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:21:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:21:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:21:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:21:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:21:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:21:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:21:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:21:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-05T22:21:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:21:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:21:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:21:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:22:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:22:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:22:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:22:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:22:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:22:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:22:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:22:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:22:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:22:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:22:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:22:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:22:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:22:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-05T22:22:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:22:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:22:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:22:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:22:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:22:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:22:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:22:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:22:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:22:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:22:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:22:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-05T22:22:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:22:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:22:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:22:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-05T22:22:03.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:03.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:03.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:22:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:22:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:22:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:22:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:22:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:22:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:22:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:22:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:22:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-05T22:22:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:22:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:22:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:22:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:22:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-05T22:22:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:22:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:22:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:22:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:22:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:22:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:22:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:22:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:22:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:22:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:22:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:22:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:22:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-05T22:22:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:22:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:22:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:22:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:22:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:22:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:22:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:22:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:22:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:22:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:22:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-05T22:22:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:22:21.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:22:21.397+0000 7f38012ac880 -1 rbd::mirror::Mirror: 0x55a8c4165d40 update_pool_replayers: restarting failed pool replayer for uuid: 8c8a266e-bd47-4f4e-af77-66406330ebb1 cluster: cluster2 client: client.mirror 2023-12-05T22:22:21.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:22:21.541+0000 7f38012ac880 -1 rbd::mirror::Mirror: 0x55a8c4165d40 update_pool_replayers: restarting failed pool replayer for uuid: a92c6d9b-946f-4025-b77d-79fb3a39d0d0 cluster: cluster2 client: client.mirror 2023-12-05T22:22:21.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:22:21.549+0000 7f3777742880 -1 rbd::mirror::Mirror: 0x564296b3dd40 update_pool_replayers: restarting failed pool replayer for uuid: 8c8a266e-bd47-4f4e-af77-66406330ebb1 cluster: cluster2 client: client.mirror 2023-12-05T22:22:21.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:22:21.553+0000 7f898003b880 -1 rbd::mirror::Mirror: 0x562f13455d40 update_pool_replayers: restarting failed pool replayer for uuid: 8c8a266e-bd47-4f4e-af77-66406330ebb1 cluster: cluster2 client: client.mirror 2023-12-05T22:22:21.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:22:21.677+0000 7f898003b880 -1 rbd::mirror::Mirror: 0x562f13455d40 update_pool_replayers: restarting failed pool replayer for uuid: a92c6d9b-946f-4025-b77d-79fb3a39d0d0 cluster: cluster2 client: client.mirror 2023-12-05T22:22:21.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:22:21.717+0000 7f8d2c763880 -1 rbd::mirror::Mirror: 0x561957dd5d40 update_pool_replayers: restarting failed pool replayer for uuid: 8c8a266e-bd47-4f4e-af77-66406330ebb1 cluster: cluster2 client: client.mirror 2023-12-05T22:22:21.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:22:21.777+0000 7f8d2c763880 -1 rbd::mirror::Mirror: 0x561957dd5d40 update_pool_replayers: restarting failed pool replayer for uuid: a92c6d9b-946f-4025-b77d-79fb3a39d0d0 cluster: cluster2 client: client.mirror 2023-12-05T22:22:21.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:22:21.849+0000 7f3777742880 -1 rbd::mirror::Mirror: 0x564296b3dd40 update_pool_replayers: restarting failed pool replayer for uuid: a92c6d9b-946f-4025-b77d-79fb3a39d0d0 cluster: cluster2 client: client.mirror 2023-12-05T22:22:23.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9277357451784112) > weight (0.25) 2023-12-05T22:22:23.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4741592442486924) > weight (0.25) 2023-12-05T22:22:23.297 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-05T22:22:23.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-05T22:22:23.298 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-05T22:22:23.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:22:23.297+0000 7f8c73d28640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 40333) UID: 1000 2023-12-05T22:22:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:22:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:22:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:22:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:22:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:22:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:22:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:22:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:22:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:22:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:22:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:22:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:22:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:22:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:22:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:22:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:22:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:22:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:22:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:22:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:22:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:22:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:22:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:22:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:22:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:22:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:22:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:22:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:22:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name= 2023-12-05T22:22:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 10 2023-12-05T22:22:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:22:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:22:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:22:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:22:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:22:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:22:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 38951936 bytes 2023-12-05T22:22:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 40157184 bytes 2023-12-05T22:22:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 63007744 bytes 2023-12-05T22:22:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 84334592 bytes 2023-12-05T22:22:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 109758464 bytes 2023-12-05T22:22:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 175526912 bytes 2023-12-05T22:22:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 233539584 bytes 2023-12-05T22:22:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 271486976 bytes 2023-12-05T22:22:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 317194240 bytes 2023-12-05T22:22:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 335769600 bytes 2023-12-05T22:22:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 349641728 bytes 2023-12-05T22:22:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 386670592 bytes 2023-12-05T22:22:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 430809088 bytes 2023-12-05T22:22:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 492686336 bytes 2023-12-05T22:22:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 541833216 bytes 2023-12-05T22:22:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 542433280 bytes 2023-12-05T22:22:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 585203712 bytes 2023-12-05T22:22:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 625585152 bytes 2023-12-05T22:22:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 663346176 bytes 2023-12-05T22:22:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 694121472 bytes 2023-12-05T22:22:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 695562240 bytes 2023-12-05T22:22:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 781291520 bytes 2023-12-05T22:22:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 822080512 bytes 2023-12-05T22:22:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 895534080 bytes 2023-12-05T22:22:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 911014912 bytes 2023-12-05T22:22:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 941625344 bytes 2023-12-05T22:22:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 959589376 bytes 2023-12-05T22:22:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 974934016 bytes 2023-12-05T22:22:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1022693376 bytes 2023-12-05T22:22:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1024645120 bytes 2023-12-05T22:22:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1065701376 bytes 2023-12-05T22:22:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1089831936 bytes 2023-12-05T22:22:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1148689408 bytes 2023-12-05T22:22:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1195845632 bytes 2023-12-05T22:22:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1270683648 bytes 2023-12-05T22:22:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1311319040 bytes 2023-12-05T22:22:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1333427200 bytes 2023-12-05T22:22:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1343211520 bytes 2023-12-05T22:22:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1353467904 bytes 2023-12-05T22:22:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1409278976 bytes 2023-12-05T22:22:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1463683072 bytes 2023-12-05T22:22:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1511813120 bytes 2023-12-05T22:22:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1533378560 bytes 2023-12-05T22:22:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1604538368 bytes 2023-12-05T22:22:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1627921408 bytes 2023-12-05T22:22:31.299 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-05T22:22:31.299 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:22:31.299 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-05T22:22:31.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-05T22:22:31.300 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-05T22:22:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1644653568 bytes 2023-12-05T22:22:31.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-05T22:22:31.302 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-05T22:22:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1686487040 bytes 2023-12-05T22:22:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1716431872 bytes 2023-12-05T22:22:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1742414848 bytes 2023-12-05T22:22:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1761103872 bytes 2023-12-05T22:22:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1802110976 bytes 2023-12-05T22:22:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1809948672 bytes 2023-12-05T22:22:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1860348928 bytes 2023-12-05T22:22:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 1877963776 bytes 2023-12-05T22:22:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 1947381760 bytes 2023-12-05T22:22:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 1972358144 bytes 2023-12-05T22:22:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2065242112 bytes 2023-12-05T22:22:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2106630144 bytes 2023-12-05T22:22:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2150121472 bytes 2023-12-05T22:22:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2206024704 bytes 2023-12-05T22:22:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2244068352 bytes 2023-12-05T22:22:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2274696192 bytes 2023-12-05T22:22:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2347271168 bytes 2023-12-05T22:22:34.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2391953408 bytes 2023-12-05T22:22:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2423300096 bytes 2023-12-05T22:22:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2462075904 bytes 2023-12-05T22:22:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2470073344 bytes 2023-12-05T22:22:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2474435584 bytes 2023-12-05T22:22:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2490131456 bytes 2023-12-05T22:22:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2504555520 bytes 2023-12-05T22:22:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2558915584 bytes 2023-12-05T22:22:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2603379712 bytes 2023-12-05T22:22:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2627225600 bytes 2023-12-05T22:22:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2627830784 bytes 2023-12-05T22:22:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2642021376 bytes 2023-12-05T22:22:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2671288320 bytes 2023-12-05T22:22:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2671889408 bytes 2023-12-05T22:22:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 2672504832 bytes 2023-12-05T22:22:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 2703790080 bytes 2023-12-05T22:22:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 2738506752 bytes 2023-12-05T22:22:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 2775281664 bytes 2023-12-05T22:22:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 2813526016 bytes 2023-12-05T22:22:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 2863273984 bytes 2023-12-05T22:22:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:22:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:22:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:22:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:22:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:22:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:22:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:22:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:22:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:22:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:22:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:22:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:22:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:22:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:22:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:22:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:22:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:22:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2023-12-05T22:22:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05T22:22:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 1 2023-12-05T22:22:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:22:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:22:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:22:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:22:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:22:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:22:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:22:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:22:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:22:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2023-12-05T22:22:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2023-12-05T22:22:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05T22:22:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:22:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:22:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2023-12-05T22:22:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2023-12-05T22:22:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05T22:22:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 1 2023-12-05T22:22:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:22:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:22:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:22:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2023-12-05T22:22:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05T22:22:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 1 2023-12-05T22:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:22:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:22:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:22:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:22:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701814917,"remote_snapshot_timestamp":1701814917,"replay_state":"idle"} 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:22:51 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:22:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:22:51 2023-12-05T22:22:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:22:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701814917,"remote_snapshot_timestamp":1701814917,"replay_state":"idle"} 2023-12-05T22:22:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:22:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:22:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:22:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:22:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:22:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:22:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:22:51 2023-12-05T22:22:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:22:51 2023-12-05T22:22:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:22:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:22:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:22:51 2023-12-05T22:22:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:22:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:22:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:22:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:22:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:22:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:22:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:22:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:22:51 2023-12-05T22:22:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:22:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:22:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:22:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:22:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap1 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:22:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2023-12-05T22:22:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:23:06.299 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.30565953484485386) > weight (0.25) 2023-12-05T22:23:06.299 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9133410543901567) > weight (0.25) 2023-12-05T22:23:06.299 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7056131172193372) > weight (0.25) 2023-12-05T22:23:06.299 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7418901267112961) > weight (0.25) 2023-12-05T22:23:06.299 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-12-05T22:23:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:23:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:23:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:23:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:23:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:23:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:23:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:23:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:23:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:23:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:23:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:23:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ localgrep cluster=cluster1 -i 2023-12-05T22:23:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:23:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:23:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:23:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:23:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:23:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:23:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:23:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:23:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:23:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:23:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:23:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:23:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:23:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:23:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:23:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:23:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:23:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:23:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:23:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:23:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:23:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:23:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:23:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:23:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:23:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:23:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:23:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:23:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:23:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:23:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:23:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:23:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:23:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:23:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:23:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:23:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:23:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:23:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:23:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:23:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:23:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:23:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:23:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:23:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:23:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:23:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:23:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 6 2023-12-05T22:23:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:23:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:23:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:23:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:23:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:23:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=6 2023-12-05T22:23:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:23:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:23:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:23:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:23:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=4 2023-12-05T22:23:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = 4 2023-12-05T22:23:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:23:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:23:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:23:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:23:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:23:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:23:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:23:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = 2023-12-05T22:23:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:23:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:23:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:23:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:23:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:23:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:23:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:23:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:23:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:23:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = 2023-12-05T22:23:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:23:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:23:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:23:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=6 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = 6 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2023-12-05T22:23:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:23:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:23:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1 2023-12-05T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2023-12-05T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:23:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:23:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:23:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:23:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:23:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap1-clean 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1-clean 2023-12-05T22:23:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:23:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:23:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:23:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2023-12-05T22:23:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:23:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:23:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 14528512 bytes 2023-12-05T22:23:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 103587840 bytes 2023-12-05T22:23:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 119471104 bytes 2023-12-05T22:23:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 120088576 bytes 2023-12-05T22:23:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 137710592 bytes 2023-12-05T22:23:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 230753280 bytes 2023-12-05T22:23:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 269996032 bytes 2023-12-05T22:23:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 316860416 bytes 2023-12-05T22:23:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 369901568 bytes 2023-12-05T22:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 388714496 bytes 2023-12-05T22:23:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 403050496 bytes 2023-12-05T22:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 403664896 bytes 2023-12-05T22:23:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 409073664 bytes 2023-12-05T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 437180416 bytes 2023-12-05T22:23:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 493706240 bytes 2023-12-05T22:23:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 565591040 bytes 2023-12-05T22:23:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 611174400 bytes 2023-12-05T22:23:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 681170944 bytes 2023-12-05T22:23:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 742953984 bytes 2023-12-05T22:23:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 794704896 bytes 2023-12-05T22:23:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 819795968 bytes 2023-12-05T22:23:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 862191616 bytes 2023-12-05T22:23:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 917772288 bytes 2023-12-05T22:23:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 933693440 bytes 2023-12-05T22:23:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 961300480 bytes 2023-12-05T22:23:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1021642752 bytes 2023-12-05T22:23:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1067844608 bytes 2023-12-05T22:23:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1108168704 bytes 2023-12-05T22:23:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1112197120 bytes 2023-12-05T22:23:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1124527104 bytes 2023-12-05T22:23:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1167875072 bytes 2023-12-05T22:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1180858368 bytes 2023-12-05T22:23:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1267040256 bytes 2023-12-05T22:23:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1324312576 bytes 2023-12-05T22:23:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1379722240 bytes 2023-12-05T22:23:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1397989376 bytes 2023-12-05T22:23:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1458731008 bytes 2023-12-05T22:23:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1514321920 bytes 2023-12-05T22:23:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1552325632 bytes 2023-12-05T22:23:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1554719744 bytes 2023-12-05T22:23:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1587978240 bytes 2023-12-05T22:23:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1588597760 bytes 2023-12-05T22:23:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1639106560 bytes 2023-12-05T22:23:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1661029376 bytes 2023-12-05T22:23:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1680155648 bytes 2023-12-05T22:23:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1753094144 bytes 2023-12-05T22:23:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1796470784 bytes 2023-12-05T22:23:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1821825024 bytes 2023-12-05T22:23:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1884106752 bytes 2023-12-05T22:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1952318464 bytes 2023-12-05T22:23:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2034826240 bytes 2023-12-05T22:23:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2054629376 bytes 2023-12-05T22:23:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2084104192 bytes 2023-12-05T22:23:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2107890688 bytes 2023-12-05T22:23:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2176655360 bytes 2023-12-05T22:23:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2189338624 bytes 2023-12-05T22:23:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2258705408 bytes 2023-12-05T22:23:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2297966592 bytes 2023-12-05T22:23:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2335806464 bytes 2023-12-05T22:23:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2399457280 bytes 2023-12-05T22:23:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2446959616 bytes 2023-12-05T22:23:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2504526848 bytes 2023-12-05T22:23:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2539728896 bytes 2023-12-05T22:23:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:23:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:23:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:23:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:23:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:23:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:23:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17889450.67,"bytes_per_snapshot":268341760.0,"last_snapshot_bytes":536683520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701814989,"remote_snapshot_timestamp":1701814989,"replay_state":"idle"} 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:23:21 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:23:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:23:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:23:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:23:21 2023-12-05T22:23:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:23:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17889450.67,"bytes_per_snapshot":268341760.0,"last_snapshot_bytes":536683520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701814989,"remote_snapshot_timestamp":1701814989,"replay_state":"idle"} 2023-12-05T22:23:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:23:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grep ^up+ 2023-12-05T22:23:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo up+replaying 2023-12-05T22:23:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:23:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:23:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:23:21 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:23:21 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:23:21 2023-12-05T22:23:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:23:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:23:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:23:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:23:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:23:28.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:23:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:23:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:23:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:23:21 2023-12-05T22:23:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:23:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:23:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:23:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:23:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap2 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2023-12-05T22:23:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:23:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:23:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:23:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:23:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:23:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:23:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:23:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:23:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:23:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:23:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:23:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:23:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:23:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:23:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:23:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:23:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:23:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:23:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:23:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:23:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:23:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:23:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:23:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:23:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:23:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:23:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:23:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:23:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:23:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:23:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:23:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:23:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:23:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:23:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:23:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:23:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:23:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:23:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:23:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:23:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:23:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:23:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:23:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:23:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:23:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:23:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:23:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:23:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:23:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:23:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:23:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:23:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:23:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:24:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 9 2023-12-05T22:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:24:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:24:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:24:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=9 2023-12-05T22:24:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:24:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:24:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:24:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:24:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = 2023-12-05T22:24:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:24:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:24:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:24:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:24:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = 2023-12-05T22:24:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:24:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:24:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:24:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:24:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = 2023-12-05T22:24:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:24:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=9 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = 9 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:24:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:24:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-05T22:24:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:24:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:24:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 4 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:24:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:24:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:24:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap1-clean ] 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1-clean 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:24:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:07.303 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5211496237229893) > weight (0.25) 2023-12-05T22:24:07.303 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:24:07.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:24:07.304 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-05T22:24:07.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:24:07.303+0000 7fa727b27640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 40331) UID: 1000 2023-12-05T22:24:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2 2023-12-05T22:24:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2023-12-05T22:24:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:24:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap2-clean 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2-clean 2023-12-05T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:24:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:24:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:24:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2023-12-05T22:24:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:24:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:24:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 23884800 bytes 2023-12-05T22:24:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 88167424 bytes 2023-12-05T22:24:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 164592640 bytes 2023-12-05T22:24:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 209637376 bytes 2023-12-05T22:24:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 301429760 bytes 2023-12-05T22:24:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 356128768 bytes 2023-12-05T22:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 405627904 bytes 2023-12-05T22:24:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 422035456 bytes 2023-12-05T22:24:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 487906304 bytes 2023-12-05T22:24:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 497804288 bytes 2023-12-05T22:24:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 572221440 bytes 2023-12-05T22:24:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 607655936 bytes 2023-12-05T22:24:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 657053696 bytes 2023-12-05T22:24:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 687215616 bytes 2023-12-05T22:24:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 730589184 bytes 2023-12-05T22:24:14.305 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:24:14.305 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:24:14.305 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:24:14.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:24:14.306 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:24:14.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:24:14.308 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-12-05T22:24:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 768929792 bytes 2023-12-05T22:24:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 785123328 bytes 2023-12-05T22:24:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 785741824 bytes 2023-12-05T22:24:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 836571136 bytes 2023-12-05T22:24:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 931106816 bytes 2023-12-05T22:24:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 980041728 bytes 2023-12-05T22:24:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1043869696 bytes 2023-12-05T22:24:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1067727872 bytes 2023-12-05T22:24:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1087014912 bytes 2023-12-05T22:24:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1135600640 bytes 2023-12-05T22:24:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1196661760 bytes 2023-12-05T22:24:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1212019712 bytes 2023-12-05T22:24:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1266999296 bytes 2023-12-05T22:24:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1320441856 bytes 2023-12-05T22:24:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1376239616 bytes 2023-12-05T22:24:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1406139392 bytes 2023-12-05T22:24:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1429354496 bytes 2023-12-05T22:24:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1485603840 bytes 2023-12-05T22:24:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:24:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:24:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:24:18.300 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8140458022647535) > weight (0.25) 2023-12-05T22:24:18.301 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-05T22:24:18.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-05T22:24:18.301 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-05T22:24:18.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:24:18.299+0000 7f8d2971e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 40231) UID: 1000 2023-12-05T22:24:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:24:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:24:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:24:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:24:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:24:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:24:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268341760.0,"last_snapshot_bytes":536683520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701814989,"remote_snapshot_timestamp":1701814989,"replay_state":"idle"} 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:23:51 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2023-12-05T22:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05T22:24:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:24:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268341760.0,"last_snapshot_bytes":536683520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701814989,"remote_snapshot_timestamp":1701814989,"replay_state":"idle"} 2023-12-05T22:24:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:24:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:24:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:24:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:24:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:24:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:24:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:23:51 2023-12-05T22:24:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:23:51 2023-12-05T22:24:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:24:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:23:51 2023-12-05T22:24:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:24:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:24:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:24:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:24:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:24:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:24:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:23:51 2023-12-05T22:24:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:24:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:24:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:24:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:24:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap3 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2023-12-05T22:24:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:24:20.301 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-05T22:24:20.302 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:24:20.302 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-05T22:24:20.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-05T22:24:20.302 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-05T22:24:20.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-05T22:24:20.305 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-12-05T22:24:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:24:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:24:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:24:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:24:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:24:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:24:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:24:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:24:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:24:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:24:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:24:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:24:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:24:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:24:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:24:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:24:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:24:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:24:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:24:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:24:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:24:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:24:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:24:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:24:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:24:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:24:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:24:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:24:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:24:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:24:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:24:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:24:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:24:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:24:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:24:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:24:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:24:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:24:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:24:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:24:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:24:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:24:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:24:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:24:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:24:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:24:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:24:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:24:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:24:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:24:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:24:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:24:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:24:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:24:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:24:49.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:24:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:24:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:24:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:24:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:24:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:24:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:24:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:24:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:24:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:24:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:24:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 13 2023-12-05T22:24:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:24:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:24:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:24:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:24:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:24:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=13 2023-12-05T22:24:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:24:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:24:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:24:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:24:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:24:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = 2023-12-05T22:24:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:24:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = 2023-12-05T22:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:24:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:24:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:24:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:24:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:24:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = 2023-12-05T22:24:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:24:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:24:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:24:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:24:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=13 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = 13 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:24:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:24:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-05T22:24:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:24:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:24:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:24:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 6 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:24:06 2023 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:24:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:24:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap2-clean ] 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2-clean 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:24:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3 2023-12-05T22:24:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:24:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:24:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2023-12-05T22:24:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:24:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:24:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:24:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:24:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap3-clean 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3-clean 2023-12-05T22:25:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:25:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:25:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:25:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2023-12-05T22:25:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:25:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:25:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 51573760 bytes 2023-12-05T22:25:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 66486272 bytes 2023-12-05T22:25:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 138109952 bytes 2023-12-05T22:25:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 164791296 bytes 2023-12-05T22:25:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 218642432 bytes 2023-12-05T22:25:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 315253760 bytes 2023-12-05T22:25:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 315856896 bytes 2023-12-05T22:25:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 337977344 bytes 2023-12-05T22:25:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 338596864 bytes 2023-12-05T22:25:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 354584576 bytes 2023-12-05T22:25:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 371968000 bytes 2023-12-05T22:25:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 402978816 bytes 2023-12-05T22:25:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 420655104 bytes 2023-12-05T22:25:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 442632192 bytes 2023-12-05T22:25:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 473417728 bytes 2023-12-05T22:25:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 501652480 bytes 2023-12-05T22:25:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 570696704 bytes 2023-12-05T22:25:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 629174272 bytes 2023-12-05T22:25:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 633989120 bytes 2023-12-05T22:25:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 656281600 bytes 2023-12-05T22:25:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 662107136 bytes 2023-12-05T22:25:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 685332480 bytes 2023-12-05T22:25:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 720289792 bytes 2023-12-05T22:25:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 754700288 bytes 2023-12-05T22:25:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 802524160 bytes 2023-12-05T22:25:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 845191168 bytes 2023-12-05T22:25:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 875932672 bytes 2023-12-05T22:25:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 985535488 bytes 2023-12-05T22:25:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1043724288 bytes 2023-12-05T22:25:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1087288320 bytes 2023-12-05T22:25:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1153716224 bytes 2023-12-05T22:25:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1200542720 bytes 2023-12-05T22:25:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1201142784 bytes 2023-12-05T22:25:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1268111360 bytes 2023-12-05T22:25:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1374132224 bytes 2023-12-05T22:25:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1374742528 bytes 2023-12-05T22:25:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1436457984 bytes 2023-12-05T22:25:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1452075008 bytes 2023-12-05T22:25:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:25:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:25:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:25:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:25:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:25:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:25:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:25:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:25:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:25:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530004480.0,"last_snapshot_bytes":523325440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815040,"remote_snapshot_timestamp":1701815040,"replay_state":"idle"} 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:24:51 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:25:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:24:51 2023-12-05T22:25:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:25:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530004480.0,"last_snapshot_bytes":523325440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815040,"remote_snapshot_timestamp":1701815040,"replay_state":"idle"} 2023-12-05T22:25:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:25:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:25:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:25:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:25:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:25:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:25:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:24:51 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:24:51 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:24:51 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:25:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:25:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:25:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:25:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:25:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:25:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:25:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:24:51 2023-12-05T22:25:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:25:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:25:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:25:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:25:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap4 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2023-12-05T22:25:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:25:19.309 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4961761164382198) > weight (0.25) 2023-12-05T22:25:19.309 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.47905057126118644) > weight (0.25) 2023-12-05T22:25:19.309 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8861874152106584) > weight (0.25) 2023-12-05T22:25:19.309 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8611876934411087) > weight (0.25) 2023-12-05T22:25:19.309 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-12-05T22:25:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:25:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:25:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:25:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:25:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:25:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:25:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:25:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:25:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:25:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:25:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:25:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:25:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:25:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:25:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:25:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:25:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:25:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:25:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:25:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:25:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:25:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:25:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:25:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:25:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:25:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:25:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:25:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:25:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:25:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:25:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:25:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:25:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:25:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:25:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:25:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:25:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:25:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:25:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:25:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:25:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:25:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:25:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:25:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:25:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:25:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:25:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:25:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:25:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:25:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:25:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:25:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 17 2023-12-05T22:25:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:25:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:25:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:25:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:25:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:25:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=17 2023-12-05T22:25:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:25:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:25:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:25:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:25:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:25:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:25:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:25:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:25:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:25:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:25:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = 2023-12-05T22:25:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:25:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:25:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:25:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:25:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:25:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:25:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:25:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:25:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:25:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = 2023-12-05T22:25:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:25:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:25:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:25:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:25:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:25:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:25:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:25:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:25:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:25:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = 2023-12-05T22:25:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:25:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:25:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:25:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=17 2023-12-05T22:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = 17 2023-12-05T22:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-05T22:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2023-12-05T22:25:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:25:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:25:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-05T22:25:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:25:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:25:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:25:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 8 2023-12-05T22:25:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:24:06 2023 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:25:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap3-clean ] 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3-clean 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:25:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:25:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:47.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5544348837459399) > weight (0.25) 2023-12-05T22:25:47.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.43147020343440645) > weight (0.25) 2023-12-05T22:25:47.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8386599034227171) > weight (0.25) 2023-12-05T22:25:47.307 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-05T22:25:47.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-05T22:25:47.307 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-05T22:25:47.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:25:47.306+0000 7f37746fd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 40236) UID: 1000 2023-12-05T22:25:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:25:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4 2023-12-05T22:25:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2023-12-05T22:25:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:25:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:25:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:25:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:25:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:25:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:25:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:25:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap4-clean 2023-12-05T22:25:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-05T22:25:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4-clean 2023-12-05T22:25:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:25:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:25:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:25:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:25:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:25:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:25:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2023-12-05T22:25:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:25:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:25:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 45501440 bytes 2023-12-05T22:25:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 86105088 bytes 2023-12-05T22:25:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 114172928 bytes 2023-12-05T22:25:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 170599424 bytes 2023-12-05T22:25:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 213083136 bytes 2023-12-05T22:25:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 213713920 bytes 2023-12-05T22:25:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 262054912 bytes 2023-12-05T22:25:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 307039232 bytes 2023-12-05T22:25:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 345457664 bytes 2023-12-05T22:25:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 424248320 bytes 2023-12-05T22:25:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 424897536 bytes 2023-12-05T22:25:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 425493504 bytes 2023-12-05T22:25:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 472715264 bytes 2023-12-05T22:25:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 501566464 bytes 2023-12-05T22:25:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 617706496 bytes 2023-12-05T22:25:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 625991680 bytes 2023-12-05T22:25:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 684673024 bytes 2023-12-05T22:25:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 748022784 bytes 2023-12-05T22:25:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 798477312 bytes 2023-12-05T22:25:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 822734848 bytes 2023-12-05T22:25:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 881457152 bytes 2023-12-05T22:25:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 950176768 bytes 2023-12-05T22:25:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 961777664 bytes 2023-12-05T22:25:55.308 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-05T22:25:55.308 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:25:55.309 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-05T22:25:55.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-05T22:25:55.309 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-05T22:25:55.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-05T22:25:55.313 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-05T22:25:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1037378560 bytes 2023-12-05T22:25:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1075468288 bytes 2023-12-05T22:25:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1133064192 bytes 2023-12-05T22:25:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1232283648 bytes 2023-12-05T22:25:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1302133760 bytes 2023-12-05T22:25:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1327400960 bytes 2023-12-05T22:25:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1368073216 bytes 2023-12-05T22:25:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1414135808 bytes 2023-12-05T22:25:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1432597504 bytes 2023-12-05T22:25:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1532786688 bytes 2023-12-05T22:25:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1533407232 bytes 2023-12-05T22:25:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1555515392 bytes 2023-12-05T22:25:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1580963840 bytes 2023-12-05T22:25:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1694910464 bytes 2023-12-05T22:25:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1731087360 bytes 2023-12-05T22:25:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1749891072 bytes 2023-12-05T22:25:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1792302080 bytes 2023-12-05T22:25:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1840557056 bytes 2023-12-05T22:25:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1889879040 bytes 2023-12-05T22:25:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1993212928 bytes 2023-12-05T22:25:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2017434624 bytes 2023-12-05T22:25:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2055902208 bytes 2023-12-05T22:25:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2086739968 bytes 2023-12-05T22:25:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2097553408 bytes 2023-12-05T22:25:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2123060224 bytes 2023-12-05T22:25:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2203052032 bytes 2023-12-05T22:25:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2226945024 bytes 2023-12-05T22:26:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2282428416 bytes 2023-12-05T22:26:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2348666880 bytes 2023-12-05T22:26:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2382627840 bytes 2023-12-05T22:26:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2447888384 bytes 2023-12-05T22:26:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2474381312 bytes 2023-12-05T22:26:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2513743872 bytes 2023-12-05T22:26:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2514653184 bytes 2023-12-05T22:26:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2533395456 bytes 2023-12-05T22:26:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2604697600 bytes 2023-12-05T22:26:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2642808832 bytes 2023-12-05T22:26:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2707000320 bytes 2023-12-05T22:26:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2730990592 bytes 2023-12-05T22:26:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2731620352 bytes 2023-12-05T22:26:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2766971904 bytes 2023-12-05T22:26:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:26:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:26:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:26:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:26:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:26:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:26:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:26:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:26:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:26:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17204701.87,"bytes_per_snapshot":520415744.0,"last_snapshot_bytes":516141056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815140,"remote_snapshot_timestamp":1701815140,"replay_state":"idle"} 2023-12-05T22:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:25:51 2023-12-05T22:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:26:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:26:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:26:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:26:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:25:51 2023-12-05T22:26:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:26:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17204701.87,"bytes_per_snapshot":520415744.0,"last_snapshot_bytes":516141056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815140,"remote_snapshot_timestamp":1701815140,"replay_state":"idle"} 2023-12-05T22:26:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:26:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:26:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:26:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:26:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:26:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:25:51 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:25:51 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:25:51 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:26:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:26:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:26:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:26:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:26:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:26:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:26:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:26:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:25:51 2023-12-05T22:26:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:26:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:26:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:26:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:26:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap5 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2023-12-05T22:26:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:26:33.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:26:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:26:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:26:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:26:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:26:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status+ mirror/test 2023-12-05T22:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:26:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:26:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:26:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:26:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:26:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:26:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:26:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:26:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:26:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:26:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:26:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:26:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:26:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:26:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:26:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:26:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:26:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:26:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:26:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:26:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:26:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:26:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:26:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:26:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:26:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:26:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:26:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:26:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:26:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:26:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:26:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:26:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:26:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:26:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:26:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:26:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:26:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + sed -e s|[/ ]|_|g 2023-12-05T22:26:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo cluster2-mirror-test.status 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:26:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 21 2023-12-05T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2+ snap list --all mirror/test --format xml 2023-12-05T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:26:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:26:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=21 2023-12-05T22:26:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:26:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:26:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:26:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:26:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:26:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:26:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:26:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:26:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:26:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:26:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = 2023-12-05T22:26:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:26:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:26:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:26:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:26:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:26:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:26:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:26:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:26:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:26:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = 2023-12-05T22:26:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:26:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:26:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:26:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:26:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:26:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:26:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:26:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:26:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:26:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = 2023-12-05T22:26:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:26:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=21 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = 21 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:26:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:26:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 10 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:26:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:26:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:25:46 2023 2023-12-05T22:26:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:26:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:26:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:26:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:26:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:26:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:26:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap4-clean ] 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4-clean 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:26:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:26:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:26:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5 2023-12-05T22:26:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2023-12-05T22:26:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:26:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:26:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:26:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:26:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:26:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap5-clean 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5-clean 2023-12-05T22:26:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:26:46.310 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6943887094981781) > weight (0.25) 2023-12-05T22:26:46.310 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:26:46.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:26:46.310 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-05T22:26:46.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:26:46.310+0000 7f85e765f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 42847) UID: 1000 2023-12-05T22:26:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:26:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:26:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:26:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:26:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:26:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 65981440 bytes 2023-12-05T22:26:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 66616320 bytes 2023-12-05T22:26:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 142492672 bytes 2023-12-05T22:26:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 204035072 bytes 2023-12-05T22:26:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 219995136 bytes 2023-12-05T22:26:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 298444800 bytes 2023-12-05T22:26:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 353788928 bytes 2023-12-05T22:26:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 381814784 bytes 2023-12-05T22:26:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 409937920 bytes 2023-12-05T22:26:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 419588096 bytes 2023-12-05T22:26:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 480718848 bytes 2023-12-05T22:26:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 501846016 bytes 2023-12-05T22:26:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 528729088 bytes 2023-12-05T22:26:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 529343488 bytes 2023-12-05T22:26:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 553828352 bytes 2023-12-05T22:26:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 559188992 bytes 2023-12-05T22:26:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 609252352 bytes 2023-12-05T22:26:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 676998144 bytes 2023-12-05T22:26:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 713712640 bytes 2023-12-05T22:26:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 736268288 bytes 2023-12-05T22:26:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 748025856 bytes 2023-12-05T22:26:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 841443328 bytes 2023-12-05T22:26:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 879177728 bytes 2023-12-05T22:26:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 898670592 bytes 2023-12-05T22:26:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 944107520 bytes 2023-12-05T22:26:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1019166720 bytes 2023-12-05T22:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1019760640 bytes 2023-12-05T22:26:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1070085120 bytes 2023-12-05T22:26:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1111067648 bytes 2023-12-05T22:26:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1118905344 bytes 2023-12-05T22:26:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1152451584 bytes 2023-12-05T22:26:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1182373888 bytes 2023-12-05T22:26:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1224982528 bytes 2023-12-05T22:26:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1312841728 bytes 2023-12-05T22:26:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1362264064 bytes 2023-12-05T22:26:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1431810048 bytes 2023-12-05T22:26:52.311 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:26:52.311 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:26:52.312 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:26:52.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:26:52.312 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:26:52.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:26:52.314 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-12-05T22:26:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1479796736 bytes 2023-12-05T22:26:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1589364736 bytes 2023-12-05T22:26:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1631421440 bytes 2023-12-05T22:26:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1651097600 bytes 2023-12-05T22:26:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1743747072 bytes 2023-12-05T22:26:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1785620480 bytes 2023-12-05T22:26:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1802592256 bytes 2023-12-05T22:26:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1840473088 bytes 2023-12-05T22:26:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1920776192 bytes 2023-12-05T22:26:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1945305088 bytes 2023-12-05T22:26:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1955021824 bytes 2023-12-05T22:26:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1967125504 bytes 2023-12-05T22:26:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1994180608 bytes 2023-12-05T22:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2065429504 bytes 2023-12-05T22:26:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2115577856 bytes 2023-12-05T22:26:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2148438016 bytes 2023-12-05T22:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2179674112 bytes 2023-12-05T22:26:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2185222144 bytes 2023-12-05T22:26:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2217764864 bytes 2023-12-05T22:26:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2271260672 bytes 2023-12-05T22:26:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2329782272 bytes 2023-12-05T22:26:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2353447936 bytes 2023-12-05T22:26:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2455208960 bytes 2023-12-05T22:26:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2485583872 bytes 2023-12-05T22:26:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2569687040 bytes 2023-12-05T22:26:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2612417536 bytes 2023-12-05T22:26:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2647269376 bytes 2023-12-05T22:26:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2738372608 bytes 2023-12-05T22:26:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2768915456 bytes 2023-12-05T22:26:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2825936896 bytes 2023-12-05T22:26:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2826541056 bytes 2023-12-05T22:26:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2881790976 bytes 2023-12-05T22:26:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2939001856 bytes 2023-12-05T22:26:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2981948416 bytes 2023-12-05T22:26:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3053051904 bytes 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:26:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:26:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:26:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:26:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:26:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:26:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:26:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17841732.27,"bytes_per_snapshot":525696512.0,"last_snapshot_bytes":535251968,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815195,"remote_snapshot_timestamp":1701815195,"replay_state":"idle"} 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:26:51 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:26:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:26:51 2023-12-05T22:26:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:26:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17841732.27,"bytes_per_snapshot":525696512.0,"last_snapshot_bytes":535251968,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815195,"remote_snapshot_timestamp":1701815195,"replay_state":"idle"} 2023-12-05T22:26:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:26:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:26:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:26:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:26:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:26:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:26:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:26:51 2023-12-05T22:26:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:26:51 2023-12-05T22:26:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:26:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:26:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:26:51 2023-12-05T22:26:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:26:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:26:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:26:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:26:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:26:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:26:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:26:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:26:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:26:51 2023-12-05T22:26:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:26:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:26:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:26:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:26:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap6 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2023-12-05T22:26:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:27:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:27:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:27:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:27:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:27:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:27:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster=cluster1 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1grep 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: -i state+ 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:local loop_instance 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:27:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:27:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:27:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:27:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:27:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:27:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:27:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:27:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:27:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:27:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:27:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:27:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:27:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:27:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:27:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:27:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:27:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:27:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:27:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:27:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:27:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:27:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:27:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:27:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:27:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:27:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:27:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:27:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:27:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:27:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:27:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:27:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:27:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:27:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:27:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:27:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:27:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 25 2023-12-05T22:27:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:27:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:27:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:27:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:27:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:27:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:27:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=25 2023-12-05T22:27:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:27:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:27:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:27:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:27:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:27:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:27:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:27:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:27:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:27:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:27:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = 2023-12-05T22:27:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:27:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:27:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = 2023-12-05T22:27:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:27:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:27:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:27:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:27:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:27:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:27:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:27:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:27:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:27:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = 2023-12-05T22:27:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:27:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:27:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:27:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:27:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:27:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=25 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = 25 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:27:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:27:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-05T22:27:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 12 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:25:46 2023 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:27:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:27:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:27:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:27:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap5-clean ] 2023-12-05T22:27:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-05T22:27:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5-clean 2023-12-05T22:27:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:27:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:27:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:27:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:27:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:27:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:27:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:27:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:41.314 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.641529173178553) > weight (0.25) 2023-12-05T22:27:41.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9991082527400938) > weight (0.25) 2023-12-05T22:27:41.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.28752022191458004) > weight (0.25) 2023-12-05T22:27:41.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3932625798522905) > weight (0.25) 2023-12-05T22:27:41.315 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-05T22:27:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:27:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap6-clean 2023-12-05T22:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-05T22:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6-clean 2023-12-05T22:27:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:27:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:27:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:27:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:27:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2023-12-05T22:27:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:27:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:27:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 25292800 bytes 2023-12-05T22:27:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 54305792 bytes 2023-12-05T22:27:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 106110976 bytes 2023-12-05T22:27:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 171864064 bytes 2023-12-05T22:27:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 201399296 bytes 2023-12-05T22:27:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 253573120 bytes 2023-12-05T22:27:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 305015808 bytes 2023-12-05T22:27:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 305631232 bytes 2023-12-05T22:27:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 357409792 bytes 2023-12-05T22:27:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 399347712 bytes 2023-12-05T22:27:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 399966208 bytes 2023-12-05T22:27:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 456969216 bytes 2023-12-05T22:27:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 480314368 bytes 2023-12-05T22:27:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 576547840 bytes 2023-12-05T22:27:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 584867840 bytes 2023-12-05T22:27:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 645447680 bytes 2023-12-05T22:27:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 695211008 bytes 2023-12-05T22:27:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 810237952 bytes 2023-12-05T22:27:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 876149760 bytes 2023-12-05T22:27:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 942315520 bytes 2023-12-05T22:27:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 972772352 bytes 2023-12-05T22:27:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1043956736 bytes 2023-12-05T22:27:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1060653056 bytes 2023-12-05T22:27:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1068939264 bytes 2023-12-05T22:27:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1139657728 bytes 2023-12-05T22:27:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1196594176 bytes 2023-12-05T22:27:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1242870784 bytes 2023-12-05T22:27:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1273545728 bytes 2023-12-05T22:27:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1345189888 bytes 2023-12-05T22:27:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1385778176 bytes 2023-12-05T22:27:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1411824640 bytes 2023-12-05T22:27:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1452747776 bytes 2023-12-05T22:27:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1516652544 bytes 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:27:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:27:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:27:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:27:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:27:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:27:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:27:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:27:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":525696512.0,"last_snapshot_bytes":535251968,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815195,"remote_snapshot_timestamp":1701815195,"replay_state":"idle"} 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:27:21 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:27:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:27:21 2023-12-05T22:27:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:27:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":525696512.0,"last_snapshot_bytes":535251968,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815195,"remote_snapshot_timestamp":1701815195,"replay_state":"idle"} 2023-12-05T22:27:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:27:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:27:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:27:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:27:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:27:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:27:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:27:21 2023-12-05T22:27:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:27:21 2023-12-05T22:27:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:27:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:27:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:27:21 2023-12-05T22:27:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:27:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:27:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:27:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:27:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:27:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:27:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:27:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:27:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:27:21 2023-12-05T22:27:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:27:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:27:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo+ up+replaying 2023-12-05T22:27:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep up+replaying 2023-12-05T22:27:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:27:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:27:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7 2023-12-05T22:27:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap7 2023-12-05T22:27:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:27:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:27:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:27:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2023-12-05T22:27:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:27:57.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.700947766022329) > weight (0.25) 2023-12-05T22:27:57.318 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:27:57.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:27:57.320 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-05T22:27:57.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:27:57.325+0000 7fc96107e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 44366) UID: 1000 2023-12-05T22:28:03.321 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:28:03.322 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:28:03.322 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:28:03.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:28:03.322 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:28:03.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:28:03.325 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-12-05T22:28:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:28:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:28:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:28:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:28:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:28:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster=cluster1 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state+ 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sleep 0 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:28:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:28:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:28:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:28:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:28:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:28:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:28:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:28:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:28:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:28:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:28:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:28:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:28:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:28:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:28:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:28:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:28:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:28:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:28:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:28:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:28:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:28:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:28:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:28:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:28:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:28:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:28:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:28:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:28:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:28:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:28:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:28:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:28:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:28:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:28:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:28:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:28:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:28:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:28:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:28:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:28:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:28:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:28:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:28:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:28:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:28:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:28:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:28:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:28:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:28:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:28:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:28:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:28:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:28:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:28:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:28:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:28:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:28:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:28:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:28:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 29 2023-12-05T22:28:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:28:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:28:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:28:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:28:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:28:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:28:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=29 2023-12-05T22:28:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:28:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:28:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:28:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:28:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:28:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:28:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:28:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:28:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:28:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = 2023-12-05T22:28:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:28:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:28:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:28:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:28:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:28:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:28:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:28:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:28:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:28:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = 2023-12-05T22:28:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:28:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:28:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:28:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:28:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:28:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:28:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:28:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = 2023-12-05T22:28:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:28:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:28:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=29 2023-12-05T22:28:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = 29 2023-12-05T22:28:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-05T22:28:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2023-12-05T22:28:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:28:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:28:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 14 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:28:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:27:38 2023 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:28:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:28:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap6-clean ] 2023-12-05T22:28:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-05T22:28:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6-clean 2023-12-05T22:28:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:28:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:28:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:28:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:28:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7 2023-12-05T22:28:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2023-12-05T22:28:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:28:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:28:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:28:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:28:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:28:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:28:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap7-clean 2023-12-05T22:28:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-05T22:28:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:28:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7-clean 2023-12-05T22:28:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:28:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2023-12-05T22:28:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:28:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:28:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 17255424 bytes 2023-12-05T22:28:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 43695104 bytes 2023-12-05T22:28:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 117825536 bytes 2023-12-05T22:28:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 158385152 bytes 2023-12-05T22:28:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 186985472 bytes 2023-12-05T22:28:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 187610112 bytes 2023-12-05T22:28:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 218378240 bytes 2023-12-05T22:28:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 246791168 bytes 2023-12-05T22:28:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 335416320 bytes 2023-12-05T22:28:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 408485888 bytes 2023-12-05T22:28:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 425839616 bytes 2023-12-05T22:28:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 437097472 bytes 2023-12-05T22:28:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 480250880 bytes 2023-12-05T22:28:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 503310336 bytes 2023-12-05T22:28:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 584396800 bytes 2023-12-05T22:28:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 651081728 bytes 2023-12-05T22:28:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 672777216 bytes 2023-12-05T22:28:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 759348224 bytes 2023-12-05T22:28:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 834653184 bytes 2023-12-05T22:28:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 867957760 bytes 2023-12-05T22:28:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 868580352 bytes 2023-12-05T22:28:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 916546560 bytes 2023-12-05T22:28:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 955318272 bytes 2023-12-05T22:28:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 974462976 bytes 2023-12-05T22:28:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1022094336 bytes 2023-12-05T22:28:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1054476288 bytes 2023-12-05T22:28:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1129091072 bytes 2023-12-05T22:28:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1171649536 bytes 2023-12-05T22:28:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1200864256 bytes 2023-12-05T22:28:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1267926016 bytes 2023-12-05T22:28:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1281994752 bytes 2023-12-05T22:28:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1338533888 bytes 2023-12-05T22:28:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1401106432 bytes 2023-12-05T22:28:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1463129088 bytes 2023-12-05T22:28:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1569085440 bytes 2023-12-05T22:28:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1591771136 bytes 2023-12-05T22:28:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1666563072 bytes 2023-12-05T22:28:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1679953920 bytes 2023-12-05T22:28:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1742407680 bytes 2023-12-05T22:28:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1801266176 bytes 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:28:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:28:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:28:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:28:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:28:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:28:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:28:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:28:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:28:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533127168.0,"last_snapshot_bytes":531002368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815251,"remote_snapshot_timestamp":1701815251,"replay_state":"idle"} 2023-12-05T22:28:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:28:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:28:21 2023-12-05T22:28:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:28:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:28:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:28:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:28:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:28:21 2023-12-05T22:28:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:28:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533127168.0,"last_snapshot_bytes":531002368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815251,"remote_snapshot_timestamp":1701815251,"replay_state":"idle"} 2023-12-05T22:28:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:28:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:28:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:28:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:28:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:28:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:28:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:28:21 2023-12-05T22:28:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:28:21 2023-12-05T22:28:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:28:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:28:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:28:21 2023-12-05T22:28:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:28:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:28:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:28:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:28:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:28:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:28:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:28:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:28:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:28:21 2023-12-05T22:28:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:28:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:28:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:28:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:28:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap8 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2023-12-05T22:28:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:28:55.316 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-05T22:28:55.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-05T22:28:55.317 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-05T22:28:55.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:28:55.320+0000 7f897cff6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 40229) UID: 1000 2023-12-05T22:28:58.317 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-05T22:28:58.318 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:28:58.318 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-05T22:28:58.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-05T22:28:58.318 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-05T22:28:58.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-05T22:28:58.321 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-05T22:29:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:29:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:29:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:29:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:29:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:29:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:29:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:29:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:29:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:29:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:29:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:29:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:29:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:29:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:29:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:29:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:29:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:29:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:29:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:29:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:29:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:29:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:29:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:29:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:29:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:29:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:29:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:29:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:29:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:29:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:29:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:29:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:29:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:29:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:29:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:29:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:29:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:29:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grep -i Replaying 2023-12-05T22:29:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo "state": "Replaying" 2023-12-05T22:29:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:29:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:29:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:29:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:29:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:29:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:29:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:29:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:29:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:29:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:29:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:29:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 33 2023-12-05T22:29:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:29:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:29:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:29:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:29:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:29:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:29:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=33 2023-12-05T22:29:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:29:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:29:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:29:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:29:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:29:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:29:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:29:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:29:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:29:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:29:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = 2023-12-05T22:29:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:29:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:29:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:29:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:29:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:29:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:29:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:29:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:29:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = 2023-12-05T22:29:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:29:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:29:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:29:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:29:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:29:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:29:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:29:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:29:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:29:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = 2023-12-05T22:29:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:29:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:29:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:29:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:29:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=33 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = 33 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:29:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:29:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-05T22:29:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:29:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:29:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:29:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 16 2023-12-05T22:29:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:27:38 2023 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:29:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:29:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap7-clean ] 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7-clean 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:29:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:29:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:29:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:29:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:29:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:29:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap8-clean 2023-12-05T22:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-05T22:29:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:29:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8-clean 2023-12-05T22:29:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:29:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:29:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:29:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:29:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2023-12-05T22:29:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:29:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:29:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 123867136 bytes 2023-12-05T22:29:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 144395264 bytes 2023-12-05T22:29:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 244683776 bytes 2023-12-05T22:29:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 323332096 bytes 2023-12-05T22:29:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 350227456 bytes 2023-12-05T22:29:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 458008576 bytes 2023-12-05T22:29:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 474276864 bytes 2023-12-05T22:29:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 479674368 bytes 2023-12-05T22:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 521233408 bytes 2023-12-05T22:29:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 576421888 bytes 2023-12-05T22:29:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 614678528 bytes 2023-12-05T22:29:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 640893952 bytes 2023-12-05T22:29:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 680108032 bytes 2023-12-05T22:29:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 729476096 bytes 2023-12-05T22:29:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 815066112 bytes 2023-12-05T22:29:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 850690048 bytes 2023-12-05T22:29:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 909256704 bytes 2023-12-05T22:29:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 930753536 bytes 2023-12-05T22:29:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 976215040 bytes 2023-12-05T22:29:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1001210880 bytes 2023-12-05T22:29:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1074894848 bytes 2023-12-05T22:29:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1078493184 bytes 2023-12-05T22:29:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1118009344 bytes 2023-12-05T22:29:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1183430656 bytes 2023-12-05T22:29:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1204231168 bytes 2023-12-05T22:29:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1238299648 bytes 2023-12-05T22:29:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1297306624 bytes 2023-12-05T22:29:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1298749440 bytes 2023-12-05T22:29:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1340027904 bytes 2023-12-05T22:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1346094080 bytes 2023-12-05T22:29:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1438115840 bytes 2023-12-05T22:29:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1470127104 bytes 2023-12-05T22:29:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1535486976 bytes 2023-12-05T22:29:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1536110592 bytes 2023-12-05T22:29:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1603097600 bytes 2023-12-05T22:29:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1679455232 bytes 2023-12-05T22:29:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1700713472 bytes 2023-12-05T22:29:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1737903104 bytes 2023-12-05T22:29:30.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.848093046719574) > weight (0.25) 2023-12-05T22:29:30.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7837814660444817) > weight (0.25) 2023-12-05T22:29:30.326 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-05T22:29:30.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-05T22:29:30.327 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-05T22:29:30.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:29:30.332+0000 7fa17eb3e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 41947) UID: 1000 2023-12-05T22:29:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1754064896 bytes 2023-12-05T22:29:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1826291712 bytes 2023-12-05T22:29:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1878881280 bytes 2023-12-05T22:29:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1899576320 bytes 2023-12-05T22:29:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1966844928 bytes 2023-12-05T22:29:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2038482944 bytes 2023-12-05T22:29:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2055581696 bytes 2023-12-05T22:29:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2103446528 bytes 2023-12-05T22:29:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2133890048 bytes 2023-12-05T22:29:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2146747392 bytes 2023-12-05T22:29:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2216854528 bytes 2023-12-05T22:29:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2267612160 bytes 2023-12-05T22:29:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2271286272 bytes 2023-12-05T22:29:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2344516608 bytes 2023-12-05T22:29:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2413295616 bytes 2023-12-05T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2447119360 bytes 2023-12-05T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2467170304 bytes 2023-12-05T22:29:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2486027264 bytes 2023-12-05T22:29:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2506171392 bytes 2023-12-05T22:29:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:29:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:29:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:29:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:29:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:29:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:29:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:29:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17438276.27,"bytes_per_snapshot":516251136.0,"last_snapshot_bytes":523148288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815352,"remote_snapshot_timestamp":1701815352,"replay_state":"idle"} 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:29:21 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:29:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:29:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:29:21 2023-12-05T22:29:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:29:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17438276.27,"bytes_per_snapshot":516251136.0,"last_snapshot_bytes":523148288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815352,"remote_snapshot_timestamp":1701815352,"replay_state":"idle"} 2023-12-05T22:29:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:29:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo up+replaying 2023-12-05T22:29:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep ^up+ 2023-12-05T22:29:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:29:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:29:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:29:21 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:29:21 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:29:21 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo+ cluster1-mirror.mirror_status 2023-12-05T22:29:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -e s|[/ ]|_|g 2023-12-05T22:29:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:29:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:29:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:29:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:29:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:29:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:29:21 2023-12-05T22:29:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap9 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2023-12-05T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:29:35.328 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-05T22:29:35.328 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:29:35.328 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-05T22:29:35.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-05T22:29:35.328 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-05T22:29:35.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-05T22:29:35.331 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2023-12-05T22:30:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:30:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:30:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ localgrep cluster=cluster1 -i 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:30:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:30:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:30:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:30:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:30:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:30:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:30:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:30:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:30:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:30:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:30:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:30:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:30:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:30:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:30:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:30:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:30:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:30:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:30:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:30:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:30:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:30:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:30:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:30:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:30:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:30:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:30:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:30:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:30:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:30:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:30:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:30:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:30:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:30:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:30:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:30:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:30:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:30:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 37 2023-12-05T22:30:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:30:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:30:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:30:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:30:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=37 2023-12-05T22:30:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:30:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:30:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:30:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:30:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:30:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:30:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:30:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:30:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:30:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = 2023-12-05T22:30:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:30:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:30:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:30:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:30:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:30:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:30:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:30:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:30:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = 2023-12-05T22:30:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:30:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:30:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:30:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:30:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:30:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:30:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:30:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:30:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = 2023-12-05T22:30:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:30:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:30:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=37 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = 37 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:30:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:30:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 18 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:30:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:29:18 2023 2023-12-05T22:30:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:30:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:30:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:30:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:30:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:30:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap8-clean ] 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8-clean 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:30:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:30:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:30:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:30:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:30:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:30:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:30:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap9-clean 2023-12-05T22:30:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-05T22:30:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9-clean 2023-12-05T22:30:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:30:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2023-12-05T22:30:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:30:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2023-12-05T22:30:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:30:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-05T22:30:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 45212672 bytes 2023-12-05T22:30:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 83614720 bytes 2023-12-05T22:30:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 135400448 bytes 2023-12-05T22:30:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 178347008 bytes 2023-12-05T22:30:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 242359296 bytes 2023-12-05T22:30:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 332928000 bytes 2023-12-05T22:30:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 398706688 bytes 2023-12-05T22:30:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 417391616 bytes 2023-12-05T22:30:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 466556928 bytes 2023-12-05T22:30:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 467166208 bytes 2023-12-05T22:30:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 556197888 bytes 2023-12-05T22:30:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 600638464 bytes 2023-12-05T22:30:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 687737856 bytes 2023-12-05T22:30:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 770213888 bytes 2023-12-05T22:30:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 817004544 bytes 2023-12-05T22:30:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 851831808 bytes 2023-12-05T22:30:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 890532864 bytes 2023-12-05T22:30:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 929270784 bytes 2023-12-05T22:30:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 962232320 bytes 2023-12-05T22:30:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 992129024 bytes 2023-12-05T22:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1014632448 bytes 2023-12-05T22:30:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1057715200 bytes 2023-12-05T22:30:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1094316032 bytes 2023-12-05T22:30:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1153465344 bytes 2023-12-05T22:30:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1191110656 bytes 2023-12-05T22:30:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1242359808 bytes 2023-12-05T22:30:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1263656960 bytes 2023-12-05T22:30:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1271308288 bytes 2023-12-05T22:30:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1291626496 bytes 2023-12-05T22:30:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1326437376 bytes 2023-12-05T22:30:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1335231488 bytes 2023-12-05T22:30:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1348199424 bytes 2023-12-05T22:30:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1372447744 bytes 2023-12-05T22:30:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1396318208 bytes 2023-12-05T22:30:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1423128576 bytes 2023-12-05T22:30:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1510582272 bytes 2023-12-05T22:30:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1577744384 bytes 2023-12-05T22:30:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1637637120 bytes 2023-12-05T22:30:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1666076672 bytes 2023-12-05T22:30:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1700999168 bytes 2023-12-05T22:30:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1733705728 bytes 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:30:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:30:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-05T22:30:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:30:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2023-12-05T22:30:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2023-12-05T22:30:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:30:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-05T22:30:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:test: 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17633075.2,"bytes_per_snapshot":526070272.0,"last_snapshot_bytes":528992256,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815406,"remote_snapshot_timestamp":1701815406,"replay_state":"idle"} 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:30:21 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2023-12-05T22:30:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2023-12-05 22:30:21 2023-12-05T22:30:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2023-12-05T22:30:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^ description: .* /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17633075.2,"bytes_per_snapshot":526070272.0,"last_snapshot_bytes":528992256,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701815406,"remote_snapshot_timestamp":1701815406,"replay_state":"idle"} 2023-12-05T22:30:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:30:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:30:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep ^up+ 2023-12-05T22:30:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:30:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep service: *mirror..* on /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2023-12-05T22:30:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.mirror_status 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update=2023-12-05 22:30:21 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-05 22:30:21 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local prev_last_update=2023-12-05 22:30:21 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror.mirror_status 2023-12-05T22:30:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:30:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror.mirror_status 2023-12-05T22:30:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-05T22:30:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:30:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2023-12-05T22:30:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-05T22:30:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update=2023-12-05 22:30:21 2023-12-05T22:30:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-05T22:30:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2023-12-05T22:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2023-12-05T22:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2023-12-05T22:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2023-12-05T22:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10 2023-12-05T22:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap10 2023-12-05T22:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2023-12-05T22:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:30:43.322 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-05T22:30:43.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-05T22:30:43.323 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-05T22:30:43.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:30:43.331+0000 7fe82fcbd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45523) UID: 1000 2023-12-05T22:30:52.324 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-05T22:30:52.324 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:30:52.325 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-05T22:30:52.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-05T22:30:52.325 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-05T22:30:52.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-05T22:30:52.328 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-05T22:30:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:30:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:30:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:30:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:30:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:30:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ greplocal instance=cluster1 -i 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state+ 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:local loop_instance 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:30:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:30:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:30:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:30:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:30:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:30:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:30:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:30:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:30:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:30:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:30:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:30:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:30:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:30:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:30:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:30:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:30:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:30:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:30:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:30:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:30:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:30:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:30:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:30:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:30:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:30:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:30:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:30:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:30:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:30:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:30:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:30:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:30:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:30:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:30:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:30:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:30:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:30:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:30:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:30:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:30:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 41 2023-12-05T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:30:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=41 2023-12-05T22:30:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:30:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:30:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:30:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:30:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:30:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:30:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:30:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:30:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:30:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = 2023-12-05T22:30:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:30:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:30:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:30:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:30:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:30:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:30:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:30:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:30:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:30:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:30:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:30:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:30:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:30:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = 2023-12-05T22:30:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:31:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:31:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:31:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:31:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:31:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:31:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:31:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:31:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:31:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = 2023-12-05T22:31:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:31:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:31:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:31:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:31:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:31:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:31:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:31:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:31:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:31:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = 2023-12-05T22:31:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:31:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:31:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:31:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:31:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=41 2023-12-05T22:31:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = 41 2023-12-05T22:31:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-05T22:31:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2023-12-05T22:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:31:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-05T22:31:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:31:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 20 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:29:18 2023 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snap9-clean ] 2023-12-05T22:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-05T22:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9-clean 2023-12-05T22:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap10 2023-12-05T22:31:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2023-12-05T22:31:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap10-clean 2023-12-05T22:31:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-05T22:31:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:31:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10-clean 2023-12-05T22:31:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-05T22:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-05T22:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-05T22:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:31:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:31:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:31:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-05T22:31:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:31:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:31:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:31:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-05T22:31:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep+ -ishift state 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:31:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:31:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:31:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:31:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:31:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:31:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:31:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:31:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:31:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:31:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-05T22:31:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:31:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:31:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:31:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-05T22:31:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:31:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:31:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:31:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:31:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:31:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-05T22:31:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-05T22:31:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:31:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:31:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-05T22:31:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:31:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:31:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:31:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:31:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:31:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:31:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:31:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:31:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-05T22:31:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:31:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:31:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:31:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:31:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:31:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:31:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-test.status 2023-12-05T22:31:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-test.status 2023-12-05T22:31:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:31:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:31:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-test.status 2023-12-05T22:31:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-test.status 2023-12-05T22:31:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:31:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-05T22:31:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:31:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-05T22:31:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 44 2023-12-05T22:31:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:31:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:31:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.status 2023-12-05T22:31:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-05T22:31:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:31:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:31:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=44 2023-12-05T22:31:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:31:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:31:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:31:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:31:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:31:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:31:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:31:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:31:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:31:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:31:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 44 = 2023-12-05T22:31:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:31:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.status 2023-12-05T22:31:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-05T22:31:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:31:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=44 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 44 = 44 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10-clean 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:31:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:31:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-05T22:31:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2023-12-05T22:31:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 21 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c53695386d 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c53695386d 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:31:06 2023 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:22:23 2023 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4d9c9d3f-bb84-4102-aaa9-f43f9335ae42 2023-12-05T22:31:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:31:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 10 2023-12-05T22:31:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1-clean 2023-12-05T22:31:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-05T22:31:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1-clean 2023-12-05T22:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:18.332 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8891910103198112) > weight (0.25) 2023-12-05T22:31:18.332 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7999879764329552) > weight (0.25) 2023-12-05T22:31:18.332 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4818123735273282) > weight (0.25) 2023-12-05T22:31:18.332 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6186824781851156) > weight (0.25) 2023-12-05T22:31:18.333 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-05T22:31:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2-clean 2023-12-05T22:31:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-05T22:31:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2-clean 2023-12-05T22:31:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:26.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3-clean 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3-clean 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4-clean 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4-clean 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5-clean 2023-12-05T22:31:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-05T22:31:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5-clean 2023-12-05T22:31:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6-clean 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6-clean 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7-clean 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7-clean 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8-clean 2023-12-05T22:31:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-05T22:31:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8-clean 2023-12-05T22:31:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9 2023-12-05T22:31:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9 2023-12-05T22:31:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2023-12-05T22:31:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9-clean 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9-clean 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap10 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10-clean 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10-clean 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:31:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export 2023-12-05T22:31:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:31:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:32:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-05T22:32:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:32:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-test.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-test.export 2023-12-05T22:32:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 10 2023-12-05T22:32:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1 2023-12-05T22:32:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-05T22:32:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap1 2023-12-05T22:32:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-05T22:32:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1-clean 2023-12-05T22:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-05T22:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap1-clean 2023-12-05T22:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-05T22:32:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2 2023-12-05T22:32:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-05T22:32:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap2 2023-12-05T22:32:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-05T22:32:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2-clean 2023-12-05T22:32:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-05T22:32:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap2-clean 2023-12-05T22:32:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-05T22:32:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3 2023-12-05T22:32:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-05T22:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap3 2023-12-05T22:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-05T22:32:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3-clean 2023-12-05T22:32:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-05T22:32:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap3-clean 2023-12-05T22:32:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-05T22:32:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4 2023-12-05T22:32:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-05T22:32:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap4 2023-12-05T22:32:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-05T22:32:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4-clean 2023-12-05T22:32:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-05T22:32:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap4-clean 2023-12-05T22:32:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-05T22:32:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5 2023-12-05T22:32:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-05T22:32:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap5 2023-12-05T22:32:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-05T22:32:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5-clean 2023-12-05T22:32:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-05T22:32:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap5-clean 2023-12-05T22:32:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-05T22:32:10.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.959361767030232) > weight (0.25) 2023-12-05T22:32:10.330 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.362671145902982) > weight (0.25) 2023-12-05T22:32:10.330 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7360909563664403) > weight (0.25) 2023-12-05T22:32:10.330 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-05T22:32:10.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-05T22:32:10.330 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-05T22:32:10.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:32:10.339+0000 7f774f531640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 43811) UID: 1000 2023-12-05T22:32:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6 2023-12-05T22:32:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-05T22:32:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap6 2023-12-05T22:32:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-05T22:32:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6-clean 2023-12-05T22:32:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-05T22:32:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap6-clean 2023-12-05T22:32:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-05T22:32:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7 2023-12-05T22:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-05T22:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap7 2023-12-05T22:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-05T22:32:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7-clean 2023-12-05T22:32:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-05T22:32:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap7-clean 2023-12-05T22:32:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-05T22:32:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8 2023-12-05T22:32:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-05T22:32:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap8 2023-12-05T22:32:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-05T22:32:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8-clean 2023-12-05T22:32:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-05T22:32:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap8-clean 2023-12-05T22:32:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-05T22:32:16.331 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-05T22:32:16.331 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:32:16.331 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-05T22:32:16.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-05T22:32:16.332 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-05T22:32:16.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-05T22:32:16.335 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-12-05T22:32:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9 2023-12-05T22:32:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-05T22:32:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap9 2023-12-05T22:32:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-05T22:32:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9-clean 2023-12-05T22:32:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-05T22:32:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap9-clean 2023-12-05T22:32:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-05T22:32:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10 2023-12-05T22:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-05T22:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap10 2023-12-05T22:32:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-05T22:32:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10-clean 2023-12-05T22:32:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-05T22:32:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap10-clean 2023-12-05T22:32:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-05T22:32:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror test 2023-12-05T22:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror test 2023-12-05T22:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-05T22:32:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-05T22:32:20.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:32:20.835+0000 7f430b50e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557049b2d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:32:20.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:32:20.835+0000 7f4311d1b640 -1 rbd::mirror::ImageReplayer: 0x557048f42a00 [3/4d9c9d3f-bb84-4102-aaa9-f43f9335ae42] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:32:21.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:32:21.487+0000 7f430b50e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557049d06340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:32:21.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:32:21.487+0000 7f4311d1b640 -1 rbd::mirror::ImageReplayer: 0x557048f42a00 [3/4d9c9d3f-bb84-4102-aaa9-f43f9335ae42] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:32:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:32:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-05T22:32:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror test 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-05T22:32:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:32:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-05T22:32:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:32:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:32:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror test deleted 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror test 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-05T22:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:32:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-05T22:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog TEST: create many images 2023-12-05T22:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.yK28jjcyo4/rbd-mirror.test.log 2023-12-05T22:32:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ date +%F %T 2023-12-05T22:32:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2023-12-05 22:32:23 TEST: create many images 2023-12-05T22:32:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2023-12-05 22:32:23 TEST: create many images 2023-12-05T22:32:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap 2023-12-05T22:32:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 50 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:32:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-05T22:32:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:32:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-05T22:32:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_1': 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1276f33f048b 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1276f33f048b 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:32:23 2023 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:32:23 2023 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:32:23 2023 2023-12-05T22:32:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:32:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:32:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6dc8b84c-1b31-4882-988d-5f4b4f42faf2 2023-12-05T22:32:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:32:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-05T22:32:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:32:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:32:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2023-12-05T22:32:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:32:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:32:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 22732800 bytes 2023-12-05T22:32:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 67764224 bytes 2023-12-05T22:32:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 146000896 bytes 2023-12-05T22:32:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 199753728 bytes 2023-12-05T22:32:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 227465216 bytes 2023-12-05T22:32:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 278358016 bytes 2023-12-05T22:32:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 287327232 bytes 2023-12-05T22:32:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 321682432 bytes 2023-12-05T22:32:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 374091776 bytes 2023-12-05T22:32:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 401283072 bytes 2023-12-05T22:32:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 429350912 bytes 2023-12-05T22:32:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 457948160 bytes 2023-12-05T22:32:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 500916224 bytes 2023-12-05T22:32:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 529329152 bytes 2023-12-05T22:32:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 612341760 bytes 2023-12-05T22:32:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 640447488 bytes 2023-12-05T22:32:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 701860864 bytes 2023-12-05T22:32:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 767265792 bytes 2023-12-05T22:32:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 788734976 bytes 2023-12-05T22:32:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 799846400 bytes 2023-12-05T22:32:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 861006848 bytes 2023-12-05T22:32:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 943685632 bytes 2023-12-05T22:32:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1003067392 bytes 2023-12-05T22:32:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1027076096 bytes 2023-12-05T22:32:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1052818432 bytes 2023-12-05T22:32:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1104692224 bytes 2023-12-05T22:32:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1158287360 bytes 2023-12-05T22:32:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1243328512 bytes 2023-12-05T22:32:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1374983168 bytes 2023-12-05T22:32:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1396771840 bytes 2023-12-05T22:32:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1463433216 bytes 2023-12-05T22:32:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1532122112 bytes 2023-12-05T22:32:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1580098560 bytes 2023-12-05T22:32:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1651685376 bytes 2023-12-05T22:32:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1680673792 bytes 2023-12-05T22:32:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1731138560 bytes 2023-12-05T22:32:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1766227968 bytes 2023-12-05T22:32:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1811992576 bytes 2023-12-05T22:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1858601984 bytes 2023-12-05T22:32:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1881483264 bytes 2023-12-05T22:32:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1963951104 bytes 2023-12-05T22:32:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1974901760 bytes 2023-12-05T22:32:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2014976000 bytes 2023-12-05T22:32:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2059462656 bytes 2023-12-05T22:32:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2108205056 bytes 2023-12-05T22:32:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2141094912 bytes 2023-12-05T22:32:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2155199488 bytes 2023-12-05T22:32:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2172080128 bytes 2023-12-05T22:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2217615360 bytes 2023-12-05T22:32:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2260443136 bytes 2023-12-05T22:32:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2269138944 bytes 2023-12-05T22:32:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2285862912 bytes 2023-12-05T22:32:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2356585472 bytes 2023-12-05T22:32:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2390857728 bytes 2023-12-05T22:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2470868992 bytes 2023-12-05T22:32:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2495644672 bytes 2023-12-05T22:32:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2579815424 bytes 2023-12-05T22:32:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2583842816 bytes 2023-12-05T22:32:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2615404544 bytes 2023-12-05T22:32:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2647195648 bytes 2023-12-05T22:32:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2734749696 bytes 2023-12-05T22:32:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:32:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-05T22:32:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:32:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-05T22:32:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-05T22:32:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_2': 2023-12-05T22:32:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:32:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:32:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:32:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 127eb0ae78ca 2023-12-05T22:32:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.127eb0ae78ca 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:32:34 2023 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:32:34 2023 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:32:34 2023 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9b0c0c99-2835-43a4-88c9-c3397c0e79b3 2023-12-05T22:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-05T22:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:32:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:32:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:32:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:32:35.333 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-05T22:32:35.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-05T22:32:35.334 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-05T22:32:35.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:32:35.342+0000 7f4312d1d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40329) UID: 1000 2023-12-05T22:32:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 65077248 bytes 2023-12-05T22:32:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 146219008 bytes 2023-12-05T22:32:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 227825664 bytes 2023-12-05T22:32:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 233625600 bytes 2023-12-05T22:32:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 247525376 bytes 2023-12-05T22:32:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 321047552 bytes 2023-12-05T22:32:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 421812224 bytes 2023-12-05T22:32:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 439908352 bytes 2023-12-05T22:32:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 516384768 bytes 2023-12-05T22:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 584244224 bytes 2023-12-05T22:32:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 608716800 bytes 2023-12-05T22:32:37.335 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-05T22:32:37.335 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:32:37.335 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-05T22:32:37.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-05T22:32:37.336 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-05T22:32:37.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-05T22:32:37.338 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-12-05T22:32:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 657266688 bytes 2023-12-05T22:32:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 751569920 bytes 2023-12-05T22:32:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 789510144 bytes 2023-12-05T22:32:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 834504704 bytes 2023-12-05T22:32:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 905259008 bytes 2023-12-05T22:32:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 956216320 bytes 2023-12-05T22:32:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 1011110912 bytes 2023-12-05T22:32:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1060232192 bytes 2023-12-05T22:32:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1126107136 bytes 2023-12-05T22:32:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1126706176 bytes 2023-12-05T22:32:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1178134528 bytes 2023-12-05T22:32:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1239399424 bytes 2023-12-05T22:32:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1293455360 bytes 2023-12-05T22:32:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1321505792 bytes 2023-12-05T22:32:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1351078912 bytes 2023-12-05T22:32:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1375618048 bytes 2023-12-05T22:32:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1425326080 bytes 2023-12-05T22:32:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1476641792 bytes 2023-12-05T22:32:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1490171904 bytes 2023-12-05T22:32:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1538751488 bytes 2023-12-05T22:32:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1539349504 bytes 2023-12-05T22:32:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1595832320 bytes 2023-12-05T22:32:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1630416896 bytes 2023-12-05T22:32:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1682141184 bytes 2023-12-05T22:32:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1713697792 bytes 2023-12-05T22:32:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1764111360 bytes 2023-12-05T22:32:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1764709376 bytes 2023-12-05T22:32:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1793252352 bytes 2023-12-05T22:32:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1835582464 bytes 2023-12-05T22:32:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1862175744 bytes 2023-12-05T22:32:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1917407232 bytes 2023-12-05T22:32:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1964860416 bytes 2023-12-05T22:32:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1995107328 bytes 2023-12-05T22:32:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1998461952 bytes 2023-12-05T22:32:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2044432384 bytes 2023-12-05T22:32:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2112829440 bytes 2023-12-05T22:32:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2160711680 bytes 2023-12-05T22:32:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2192890880 bytes 2023-12-05T22:32:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2218039296 bytes 2023-12-05T22:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2231988224 bytes 2023-12-05T22:32:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2300063744 bytes 2023-12-05T22:32:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2318873600 bytes 2023-12-05T22:32:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2395831296 bytes 2023-12-05T22:32:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2410178560 bytes 2023-12-05T22:32:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2507777024 bytes 2023-12-05T22:32:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2518499328 bytes 2023-12-05T22:32:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2585096192 bytes 2023-12-05T22:32:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2589378560 bytes 2023-12-05T22:32:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2626279424 bytes 2023-12-05T22:32:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2626870272 bytes 2023-12-05T22:32:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2647052288 bytes 2023-12-05T22:32:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2651900928 bytes 2023-12-05T22:32:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2687295488 bytes 2023-12-05T22:32:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2736022528 bytes 2023-12-05T22:32:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2769554432 bytes 2023-12-05T22:32:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2861642752 bytes 2023-12-05T22:32:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2902418432 bytes 2023-12-05T22:32:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2952819712 bytes 2023-12-05T22:32:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2989547520 bytes 2023-12-05T22:32:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3043044352 bytes 2023-12-05T22:32:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:32:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-05T22:32:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:32:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:32:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-05T22:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-05T22:32:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:32:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-05T22:32:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_3': 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12903fcd5337 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12903fcd5337 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:32:47 2023 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:32:47 2023 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:32:47 2023 2023-12-05T22:32:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:32:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:32:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 09ba1f5d-4c90-48e3-a545-11284c264b5a 2023-12-05T22:32:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:32:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:32:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-05T22:32:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:32:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:32:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:32:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:32:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:32:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:32:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:32:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 18614272 bytes 2023-12-05T22:32:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 42299392 bytes 2023-12-05T22:32:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 75905024 bytes 2023-12-05T22:32:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 150108160 bytes 2023-12-05T22:32:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 173265920 bytes 2023-12-05T22:32:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 210084864 bytes 2023-12-05T22:32:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 295715840 bytes 2023-12-05T22:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 327249920 bytes 2023-12-05T22:32:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 340054016 bytes 2023-12-05T22:32:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 347046912 bytes 2023-12-05T22:32:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 395815936 bytes 2023-12-05T22:32:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 445147136 bytes 2023-12-05T22:32:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 548601856 bytes 2023-12-05T22:32:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 587838464 bytes 2023-12-05T22:32:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 639267840 bytes 2023-12-05T22:32:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 639879168 bytes 2023-12-05T22:32:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 668467200 bytes 2023-12-05T22:32:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 688695296 bytes 2023-12-05T22:32:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 745867264 bytes 2023-12-05T22:32:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 823093248 bytes 2023-12-05T22:32:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 852989952 bytes 2023-12-05T22:32:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 887655424 bytes 2023-12-05T22:32:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 928730112 bytes 2023-12-05T22:32:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 957882368 bytes 2023-12-05T22:32:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 999869440 bytes 2023-12-05T22:32:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1017207808 bytes 2023-12-05T22:32:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1039598592 bytes 2023-12-05T22:32:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1131258880 bytes 2023-12-05T22:32:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1202356224 bytes 2023-12-05T22:32:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1235182592 bytes 2023-12-05T22:32:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1242710016 bytes 2023-12-05T22:32:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1255808000 bytes 2023-12-05T22:32:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1276954624 bytes 2023-12-05T22:32:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1348327424 bytes 2023-12-05T22:32:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1374932992 bytes 2023-12-05T22:32:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1423626240 bytes 2023-12-05T22:32:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1498753024 bytes 2023-12-05T22:32:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1536788480 bytes 2023-12-05T22:32:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1563770880 bytes 2023-12-05T22:32:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1637334016 bytes 2023-12-05T22:32:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1721943040 bytes 2023-12-05T22:32:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1763329024 bytes 2023-12-05T22:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1786144768 bytes 2023-12-05T22:32:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1837531136 bytes 2023-12-05T22:32:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1882194944 bytes 2023-12-05T22:32:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1912783872 bytes 2023-12-05T22:32:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1962619904 bytes 2023-12-05T22:32:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2010514432 bytes 2023-12-05T22:32:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2052892672 bytes 2023-12-05T22:32:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2073985024 bytes 2023-12-05T22:32:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2105556992 bytes 2023-12-05T22:32:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2115671040 bytes 2023-12-05T22:32:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2131382272 bytes 2023-12-05T22:32:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2154614784 bytes 2023-12-05T22:32:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2227566592 bytes 2023-12-05T22:32:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2228182016 bytes 2023-12-05T22:32:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2251602944 bytes 2023-12-05T22:32:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2316674048 bytes 2023-12-05T22:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2351116288 bytes 2023-12-05T22:32:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2426526720 bytes 2023-12-05T22:32:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2542767104 bytes 2023-12-05T22:32:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2557288448 bytes 2023-12-05T22:32:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2661731328 bytes 2023-12-05T22:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2695558144 bytes 2023-12-05T22:32:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2824374272 bytes 2023-12-05T22:32:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2837498880 bytes 2023-12-05T22:32:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2873262080 bytes 2023-12-05T22:32:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2902802432 bytes 2023-12-05T22:32:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2956179456 bytes 2023-12-05T22:32:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2975372288 bytes 2023-12-05T22:33:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2994907136 bytes 2023-12-05T22:33:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3067360256 bytes 2023-12-05T22:33:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3089669120 bytes 2023-12-05T22:33:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3130602496 bytes 2023-12-05T22:33:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3212173312 bytes 2023-12-05T22:33:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3268632576 bytes 2023-12-05T22:33:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:33:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:33:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:33:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2023-12-05T22:33:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-05T22:33:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:33:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:33:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-05T22:33:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:33:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-05T22:33:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:33:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_4': 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129976251ac7 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129976251ac7 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:33:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:01 2023 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:01 2023 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:01 2023 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: aa119937-ee52-43d0-81c6-311c63b77e8f 2023-12-05T22:33:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:33:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:33:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-05T22:33:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:33:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:33:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:33:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:33:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:33:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 77890560 bytes 2023-12-05T22:33:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 80034816 bytes 2023-12-05T22:33:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 184919040 bytes 2023-12-05T22:33:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 185526272 bytes 2023-12-05T22:33:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 186146816 bytes 2023-12-05T22:33:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 265239552 bytes 2023-12-05T22:33:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 290105344 bytes 2023-12-05T22:33:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 364855296 bytes 2023-12-05T22:33:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 423617536 bytes 2023-12-05T22:33:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 430429184 bytes 2023-12-05T22:33:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 518618112 bytes 2023-12-05T22:33:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 569755648 bytes 2023-12-05T22:33:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 592993280 bytes 2023-12-05T22:33:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 629096448 bytes 2023-12-05T22:33:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 648068096 bytes 2023-12-05T22:33:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 698664960 bytes 2023-12-05T22:33:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 724299776 bytes 2023-12-05T22:33:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 771719168 bytes 2023-12-05T22:33:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 823914496 bytes 2023-12-05T22:33:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 880839680 bytes 2023-12-05T22:33:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1008006144 bytes 2023-12-05T22:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1022678016 bytes 2023-12-05T22:33:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1027408896 bytes 2023-12-05T22:33:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1045164032 bytes 2023-12-05T22:33:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1106208768 bytes 2023-12-05T22:33:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1129067520 bytes 2023-12-05T22:33:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1162783744 bytes 2023-12-05T22:33:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1172150272 bytes 2023-12-05T22:33:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1204342784 bytes 2023-12-05T22:33:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1211161600 bytes 2023-12-05T22:33:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1262699520 bytes 2023-12-05T22:33:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1319845888 bytes 2023-12-05T22:33:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1345539072 bytes 2023-12-05T22:33:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1357689856 bytes 2023-12-05T22:33:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1429707776 bytes 2023-12-05T22:33:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1462682624 bytes 2023-12-05T22:33:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1539375104 bytes 2023-12-05T22:33:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1559812096 bytes 2023-12-05T22:33:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1582493696 bytes 2023-12-05T22:33:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1588346880 bytes 2023-12-05T22:33:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1593990144 bytes 2023-12-05T22:33:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1594590208 bytes 2023-12-05T22:33:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1660022784 bytes 2023-12-05T22:33:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1722668032 bytes 2023-12-05T22:33:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1767071744 bytes 2023-12-05T22:33:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1899977728 bytes 2023-12-05T22:33:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1959250944 bytes 2023-12-05T22:33:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1967722496 bytes 2023-12-05T22:33:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2052331520 bytes 2023-12-05T22:33:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2070194176 bytes 2023-12-05T22:33:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2092244992 bytes 2023-12-05T22:33:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2136215552 bytes 2023-12-05T22:33:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2172290048 bytes 2023-12-05T22:33:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2187654144 bytes 2023-12-05T22:33:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2228583424 bytes 2023-12-05T22:33:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2245295104 bytes 2023-12-05T22:33:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2315192320 bytes 2023-12-05T22:33:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2321580032 bytes 2023-12-05T22:33:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2357872640 bytes 2023-12-05T22:33:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2373362688 bytes 2023-12-05T22:33:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2405874688 bytes 2023-12-05T22:33:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2451214336 bytes 2023-12-05T22:33:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2530492416 bytes 2023-12-05T22:33:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2544116736 bytes 2023-12-05T22:33:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2623770624 bytes 2023-12-05T22:33:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2689589248 bytes 2023-12-05T22:33:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2746549248 bytes 2023-12-05T22:33:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2774299648 bytes 2023-12-05T22:33:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2787250176 bytes 2023-12-05T22:33:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2835932160 bytes 2023-12-05T22:33:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2836583424 bytes 2023-12-05T22:33:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2837188608 bytes 2023-12-05T22:33:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2870522880 bytes 2023-12-05T22:33:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2889659392 bytes 2023-12-05T22:33:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2906385408 bytes 2023-12-05T22:33:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3003150336 bytes 2023-12-05T22:33:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3046740992 bytes 2023-12-05T22:33:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3129591808 bytes 2023-12-05T22:33:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3179768832 bytes 2023-12-05T22:33:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3236318208 bytes 2023-12-05T22:33:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3250449408 bytes 2023-12-05T22:33:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3325714432 bytes 2023-12-05T22:33:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3379169280 bytes 2023-12-05T22:33:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3385934848 bytes 2023-12-05T22:33:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3432915968 bytes 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-05T22:33:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-05T22:33:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_5': 2023-12-05T22:33:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:33:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:33:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:33:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12a1148a9492 2023-12-05T22:33:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12a1148a9492 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:15 2023 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:15 2023 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:15 2023 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 23aac29b-db31-4b69-bed6-9ec5773f48c0 2023-12-05T22:33:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:33:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-05T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:33:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:33:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2023-12-05T22:33:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:33:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:33:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 38387712 bytes 2023-12-05T22:33:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 107726848 bytes 2023-12-05T22:33:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 143639552 bytes 2023-12-05T22:33:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 227929088 bytes 2023-12-05T22:33:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 287912960 bytes 2023-12-05T22:33:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 346512384 bytes 2023-12-05T22:33:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 435540992 bytes 2023-12-05T22:33:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 446709760 bytes 2023-12-05T22:33:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 485001216 bytes 2023-12-05T22:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 530584576 bytes 2023-12-05T22:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 589533184 bytes 2023-12-05T22:33:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 607410176 bytes 2023-12-05T22:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 632628224 bytes 2023-12-05T22:33:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 683025408 bytes 2023-12-05T22:33:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 725690368 bytes 2023-12-05T22:33:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 786102272 bytes 2023-12-05T22:33:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 786721792 bytes 2023-12-05T22:33:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 857909248 bytes 2023-12-05T22:33:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 869865472 bytes 2023-12-05T22:33:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 904935424 bytes 2023-12-05T22:33:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 921190400 bytes 2023-12-05T22:33:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 926848000 bytes 2023-12-05T22:33:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 971834368 bytes 2023-12-05T22:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1045492736 bytes 2023-12-05T22:33:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1058546688 bytes 2023-12-05T22:33:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1138299904 bytes 2023-12-05T22:33:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1163253760 bytes 2023-12-05T22:33:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1243155456 bytes 2023-12-05T22:33:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1304308736 bytes 2023-12-05T22:33:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1322574848 bytes 2023-12-05T22:33:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1335216128 bytes 2023-12-05T22:33:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1419493376 bytes 2023-12-05T22:33:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1492329472 bytes 2023-12-05T22:33:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1533182976 bytes 2023-12-05T22:33:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1547556864 bytes 2023-12-05T22:33:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1595793408 bytes 2023-12-05T22:33:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1698272256 bytes 2023-12-05T22:33:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1737219072 bytes 2023-12-05T22:33:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1761211392 bytes 2023-12-05T22:33:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1854481408 bytes 2023-12-05T22:33:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1892179968 bytes 2023-12-05T22:33:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1945745408 bytes 2023-12-05T22:33:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:33:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:33:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-05T22:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-05T22:33:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:33:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:33:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-05T22:33:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:33:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-05T22:33:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:33:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_6': 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ab73347634 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ab73347634 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:24 2023 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:24 2023 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:24 2023 2023-12-05T22:33:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:33:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:33:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ee1e6e37-e0d9-4a81-a633-71147ddff486 2023-12-05T22:33:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:33:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:33:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-05T22:33:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:33:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:33:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:33:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:33:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:33:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 59815936 bytes 2023-12-05T22:33:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 85509120 bytes 2023-12-05T22:33:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 110346240 bytes 2023-12-05T22:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 151781376 bytes 2023-12-05T22:33:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 185128960 bytes 2023-12-05T22:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 244818944 bytes 2023-12-05T22:33:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 265993216 bytes 2023-12-05T22:33:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 290118656 bytes 2023-12-05T22:33:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 358795264 bytes 2023-12-05T22:33:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 366559232 bytes 2023-12-05T22:33:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 404455424 bytes 2023-12-05T22:33:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 456563712 bytes 2023-12-05T22:33:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 481444864 bytes 2023-12-05T22:33:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 612754432 bytes 2023-12-05T22:33:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 659862528 bytes 2023-12-05T22:33:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 707919872 bytes 2023-12-05T22:33:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 755356672 bytes 2023-12-05T22:33:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 823929856 bytes 2023-12-05T22:33:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 847729664 bytes 2023-12-05T22:33:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 945425408 bytes 2023-12-05T22:33:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1007717376 bytes 2023-12-05T22:33:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1057444864 bytes 2023-12-05T22:33:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1092599808 bytes 2023-12-05T22:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1115871232 bytes 2023-12-05T22:33:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1184264192 bytes 2023-12-05T22:33:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1226757120 bytes 2023-12-05T22:33:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1282797568 bytes 2023-12-05T22:33:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1366700032 bytes 2023-12-05T22:33:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1367337984 bytes 2023-12-05T22:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1406754816 bytes 2023-12-05T22:33:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1428338688 bytes 2023-12-05T22:33:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1439934464 bytes 2023-12-05T22:33:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1527124992 bytes 2023-12-05T22:33:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1573382144 bytes 2023-12-05T22:33:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1574005760 bytes 2023-12-05T22:33:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1640859648 bytes 2023-12-05T22:33:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1717650432 bytes 2023-12-05T22:33:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1795398656 bytes 2023-12-05T22:33:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1851069440 bytes 2023-12-05T22:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1890722816 bytes 2023-12-05T22:33:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1907601408 bytes 2023-12-05T22:33:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1945478144 bytes 2023-12-05T22:33:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1971735552 bytes 2023-12-05T22:33:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2004191232 bytes 2023-12-05T22:33:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2015922176 bytes 2023-12-05T22:33:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2028730368 bytes 2023-12-05T22:33:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2058759168 bytes 2023-12-05T22:33:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2108944384 bytes 2023-12-05T22:33:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2137597952 bytes 2023-12-05T22:33:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2157602816 bytes 2023-12-05T22:33:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2221420544 bytes 2023-12-05T22:33:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2246764544 bytes 2023-12-05T22:33:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2330435584 bytes 2023-12-05T22:33:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2355919872 bytes 2023-12-05T22:33:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2407519232 bytes 2023-12-05T22:33:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2434962432 bytes 2023-12-05T22:33:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2482800640 bytes 2023-12-05T22:33:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2559897600 bytes 2023-12-05T22:33:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2578958336 bytes 2023-12-05T22:33:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2633443328 bytes 2023-12-05T22:33:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2680872960 bytes 2023-12-05T22:33:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2745663488 bytes 2023-12-05T22:33:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2773319680 bytes 2023-12-05T22:33:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2792471552 bytes 2023-12-05T22:33:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2799305728 bytes 2023-12-05T22:33:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2830457856 bytes 2023-12-05T22:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2903292928 bytes 2023-12-05T22:33:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2913265664 bytes 2023-12-05T22:33:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2982233088 bytes 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:33:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-05T22:33:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:33:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:33:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-05T22:33:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:33:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-05T22:33:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:33:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-05T22:33:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_7': 2023-12-05T22:33:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:33:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:33:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:33:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12b48b121999 2023-12-05T22:33:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12b48b121999 2023-12-05T22:33:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:37 2023 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:37 2023 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:37 2023 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 01c4381c-2953-4889-8d37-61725def0e37 2023-12-05T22:33:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:33:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:33:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-05T22:33:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:33:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:33:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2023-12-05T22:33:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:33:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:33:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 32364544 bytes 2023-12-05T22:33:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 54139904 bytes 2023-12-05T22:33:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 99807232 bytes 2023-12-05T22:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 169699328 bytes 2023-12-05T22:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 213669888 bytes 2023-12-05T22:33:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 233665536 bytes 2023-12-05T22:33:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 298000384 bytes 2023-12-05T22:33:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 332576768 bytes 2023-12-05T22:33:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 355420160 bytes 2023-12-05T22:33:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 389645312 bytes 2023-12-05T22:33:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 422217728 bytes 2023-12-05T22:33:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 496990208 bytes 2023-12-05T22:33:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 555294720 bytes 2023-12-05T22:33:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 602172416 bytes 2023-12-05T22:33:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 645089280 bytes 2023-12-05T22:33:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 723026944 bytes 2023-12-05T22:33:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 749220864 bytes 2023-12-05T22:33:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 774775808 bytes 2023-12-05T22:33:40.336 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.30058231480811837) > weight (0.25) 2023-12-05T22:33:40.336 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9925905631458936) > weight (0.25) 2023-12-05T22:33:40.336 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-05T22:33:40.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-05T22:33:40.336 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-05T22:33:40.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:33:40.345+0000 7f37fe267640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40233) UID: 1000 2023-12-05T22:33:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 856484864 bytes 2023-12-05T22:33:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 867859456 bytes 2023-12-05T22:33:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 922920960 bytes 2023-12-05T22:33:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 970901504 bytes 2023-12-05T22:33:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1023750144 bytes 2023-12-05T22:33:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1055033344 bytes 2023-12-05T22:33:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1065398272 bytes 2023-12-05T22:33:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1091565568 bytes 2023-12-05T22:33:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1122759680 bytes 2023-12-05T22:33:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1256265728 bytes 2023-12-05T22:33:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1282874368 bytes 2023-12-05T22:33:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1306221568 bytes 2023-12-05T22:33:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1329497088 bytes 2023-12-05T22:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1360015360 bytes 2023-12-05T22:33:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1416207360 bytes 2023-12-05T22:33:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1495320576 bytes 2023-12-05T22:33:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1495949312 bytes 2023-12-05T22:33:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1541655552 bytes 2023-12-05T22:33:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1603097600 bytes 2023-12-05T22:33:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1618745344 bytes 2023-12-05T22:33:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1639596032 bytes 2023-12-05T22:33:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1698721792 bytes 2023-12-05T22:33:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1727591424 bytes 2023-12-05T22:33:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1792941056 bytes 2023-12-05T22:33:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1824275456 bytes 2023-12-05T22:33:44.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1896999936 bytes 2023-12-05T22:33:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1922384896 bytes 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:33:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:33:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:33:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-05T22:33:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-05T22:33:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:33:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-05T22:33:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_8': 2023-12-05T22:33:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12bec6189f1d 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12bec6189f1d 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:45 2023 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:45 2023 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:45 2023 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 5e8d2584-d40c-4e8c-bda1-2d1342a10e52 2023-12-05T22:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-05T22:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:33:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2023-12-05T22:33:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:33:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:33:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 18545664 bytes 2023-12-05T22:33:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 99604480 bytes 2023-12-05T22:33:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 144069632 bytes 2023-12-05T22:33:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 171192320 bytes 2023-12-05T22:33:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 294558720 bytes 2023-12-05T22:33:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 337244160 bytes 2023-12-05T22:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 378977280 bytes 2023-12-05T22:33:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 445723648 bytes 2023-12-05T22:33:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 483173376 bytes 2023-12-05T22:33:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 550043648 bytes 2023-12-05T22:33:48.337 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-05T22:33:48.337 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:33:48.338 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-05T22:33:48.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-05T22:33:48.338 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-05T22:33:48.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-05T22:33:48.341 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-05T22:33:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 586575872 bytes 2023-12-05T22:33:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 606095360 bytes 2023-12-05T22:33:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 613827584 bytes 2023-12-05T22:33:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 649122816 bytes 2023-12-05T22:33:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 666091520 bytes 2023-12-05T22:33:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 673072128 bytes 2023-12-05T22:33:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 755921920 bytes 2023-12-05T22:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 788983808 bytes 2023-12-05T22:33:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 798000128 bytes 2023-12-05T22:33:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 868789248 bytes 2023-12-05T22:33:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 895695872 bytes 2023-12-05T22:33:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 919787520 bytes 2023-12-05T22:33:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 941452288 bytes 2023-12-05T22:33:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 962353152 bytes 2023-12-05T22:33:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1032680448 bytes 2023-12-05T22:33:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1072965632 bytes 2023-12-05T22:33:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1086586880 bytes 2023-12-05T22:33:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1114120192 bytes 2023-12-05T22:33:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1184427008 bytes 2023-12-05T22:33:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1230602240 bytes 2023-12-05T22:33:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1297261568 bytes 2023-12-05T22:33:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1382496256 bytes 2023-12-05T22:33:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1433847808 bytes 2023-12-05T22:33:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1508188160 bytes 2023-12-05T22:33:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1556965376 bytes 2023-12-05T22:33:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1580846080 bytes 2023-12-05T22:33:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1584877568 bytes 2023-12-05T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1609912320 bytes 2023-12-05T22:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1646141440 bytes 2023-12-05T22:33:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1689296896 bytes 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-05T22:33:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:33:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-05T22:33:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_9': 2023-12-05T22:33:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:33:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:33:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:33:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ce714792b3 2023-12-05T22:33:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ce714792b3 2023-12-05T22:33:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:33:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:33:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:33:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:33:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:53 2023 2023-12-05T22:33:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:53 2023 2023-12-05T22:33:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:53 2023 2023-12-05T22:33:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:33:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:33:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d0e14dbe-197f-4ab0-8cb5-c1acd3e96372 2023-12-05T22:33:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:33:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:33:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-05T22:33:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:33:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:33:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2023-12-05T22:33:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:33:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:33:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 43943936 bytes 2023-12-05T22:33:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 75357184 bytes 2023-12-05T22:33:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 120757248 bytes 2023-12-05T22:33:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 234731520 bytes 2023-12-05T22:33:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 285193216 bytes 2023-12-05T22:33:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 353972224 bytes 2023-12-05T22:33:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 433130496 bytes 2023-12-05T22:33:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 476311552 bytes 2023-12-05T22:33:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 556881920 bytes 2023-12-05T22:33:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 567410688 bytes 2023-12-05T22:33:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 603757568 bytes 2023-12-05T22:33:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 622466048 bytes 2023-12-05T22:33:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 711532544 bytes 2023-12-05T22:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 777886720 bytes 2023-12-05T22:33:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 823857152 bytes 2023-12-05T22:33:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 874735616 bytes 2023-12-05T22:33:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 894790656 bytes 2023-12-05T22:33:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 898937856 bytes 2023-12-05T22:33:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 933657600 bytes 2023-12-05T22:33:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 983035904 bytes 2023-12-05T22:33:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1038025728 bytes 2023-12-05T22:33:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1059411968 bytes 2023-12-05T22:33:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1069880320 bytes 2023-12-05T22:33:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1122796544 bytes 2023-12-05T22:33:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1190778880 bytes 2023-12-05T22:33:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1216210944 bytes 2023-12-05T22:33:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1265159168 bytes 2023-12-05T22:33:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1304261632 bytes 2023-12-05T22:33:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1347422208 bytes 2023-12-05T22:33:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1373309952 bytes 2023-12-05T22:33:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1410400256 bytes 2023-12-05T22:33:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1491806208 bytes 2023-12-05T22:33:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:33:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-05T22:33:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:33:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:33:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:33:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-05T22:33:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:33:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:33:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:33:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:33:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-05T22:34:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:34:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-05T22:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_10': 2023-12-05T22:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12d6838136b8 2023-12-05T22:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12d6838136b8 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:59 2023 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:59 2023 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:59 2023 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 0042551e-e667-4458-be32-9094ff2c9220 2023-12-05T22:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-05T22:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:34:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:34:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2023-12-05T22:34:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:34:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:34:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 28805120 bytes 2023-12-05T22:34:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 46601216 bytes 2023-12-05T22:34:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 70257664 bytes 2023-12-05T22:34:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 70863872 bytes 2023-12-05T22:34:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 131367936 bytes 2023-12-05T22:34:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 158770176 bytes 2023-12-05T22:34:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 191605760 bytes 2023-12-05T22:34:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 267907072 bytes 2023-12-05T22:34:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 279662592 bytes 2023-12-05T22:34:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 333455360 bytes 2023-12-05T22:34:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 410480640 bytes 2023-12-05T22:34:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 468966400 bytes 2023-12-05T22:34:02.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 505214976 bytes 2023-12-05T22:34:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 540016640 bytes 2023-12-05T22:34:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 550149120 bytes 2023-12-05T22:34:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 574058496 bytes 2023-12-05T22:34:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 648726528 bytes 2023-12-05T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 688977920 bytes 2023-12-05T22:34:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 717894656 bytes 2023-12-05T22:34:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 763296768 bytes 2023-12-05T22:34:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 810475520 bytes 2023-12-05T22:34:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 829654016 bytes 2023-12-05T22:34:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 830268416 bytes 2023-12-05T22:34:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 850570240 bytes 2023-12-05T22:34:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 851188736 bytes 2023-12-05T22:34:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 861162496 bytes 2023-12-05T22:34:04.339 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.929674996404649) > weight (0.25) 2023-12-05T22:34:04.339 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:34:04.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:34:04.340 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-05T22:34:04.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:34:04.348+0000 7fdfe6b07640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 44960) UID: 1000 2023-12-05T22:34:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 875542528 bytes 2023-12-05T22:34:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 925176832 bytes 2023-12-05T22:34:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 985927680 bytes 2023-12-05T22:34:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 997760000 bytes 2023-12-05T22:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1022990336 bytes 2023-12-05T22:34:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1134342144 bytes 2023-12-05T22:34:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1189810176 bytes 2023-12-05T22:34:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1293623296 bytes 2023-12-05T22:34:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1320851456 bytes 2023-12-05T22:34:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1367607296 bytes 2023-12-05T22:34:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1389217792 bytes 2023-12-05T22:34:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1419724800 bytes 2023-12-05T22:34:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1473269760 bytes 2023-12-05T22:34:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1529320448 bytes 2023-12-05T22:34:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1577595904 bytes 2023-12-05T22:34:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1639767040 bytes 2023-12-05T22:34:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1688229888 bytes 2023-12-05T22:34:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1688832000 bytes 2023-12-05T22:34:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1746367488 bytes 2023-12-05T22:34:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1785347072 bytes 2023-12-05T22:34:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1840023552 bytes 2023-12-05T22:34:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1865634816 bytes 2023-12-05T22:34:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1876651008 bytes 2023-12-05T22:34:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1950333952 bytes 2023-12-05T22:34:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2020406272 bytes 2023-12-05T22:34:09.341 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:34:09.341 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:34:09.341 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:34:09.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:34:09.341 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:34:09.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:34:09.343 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-05T22:34:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2054404096 bytes 2023-12-05T22:34:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2065385472 bytes 2023-12-05T22:34:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2100211712 bytes 2023-12-05T22:34:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2219514880 bytes 2023-12-05T22:34:10.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2231919616 bytes 2023-12-05T22:34:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2289574912 bytes 2023-12-05T22:34:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2301925376 bytes 2023-12-05T22:34:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2351141888 bytes 2023-12-05T22:34:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2396261376 bytes 2023-12-05T22:34:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2439813120 bytes 2023-12-05T22:34:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:34:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:34:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:34:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:34:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:34:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:34:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:34:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2023-12-05T22:34:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:34:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-05T22:34:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:34:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_11': 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12e855263319 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12e855263319 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:34:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:11 2023 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:11 2023 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:11 2023 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 08d57849-e22c-4949-ad8e-26b3cce133fe 2023-12-05T22:34:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:34:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:34:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-05T22:34:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:34:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:34:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2023-12-05T22:34:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:34:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:34:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 33779712 bytes 2023-12-05T22:34:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 74248192 bytes 2023-12-05T22:34:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 125737984 bytes 2023-12-05T22:34:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 153976832 bytes 2023-12-05T22:34:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 198177792 bytes 2023-12-05T22:34:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 198772736 bytes 2023-12-05T22:34:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 308531200 bytes 2023-12-05T22:34:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 309141504 bytes 2023-12-05T22:34:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 345729024 bytes 2023-12-05T22:34:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 417676288 bytes 2023-12-05T22:34:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 515756032 bytes 2023-12-05T22:34:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 532658176 bytes 2023-12-05T22:34:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 543267840 bytes 2023-12-05T22:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 589777920 bytes 2023-12-05T22:34:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 645685248 bytes 2023-12-05T22:34:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 685629440 bytes 2023-12-05T22:34:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 733367296 bytes 2023-12-05T22:34:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 788569088 bytes 2023-12-05T22:34:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 838852608 bytes 2023-12-05T22:34:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 841353216 bytes 2023-12-05T22:34:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 893546496 bytes 2023-12-05T22:34:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 996599808 bytes 2023-12-05T22:34:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1012582400 bytes 2023-12-05T22:34:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1063255040 bytes 2023-12-05T22:34:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1162271744 bytes 2023-12-05T22:34:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1181402112 bytes 2023-12-05T22:34:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1187914752 bytes 2023-12-05T22:34:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1211699200 bytes 2023-12-05T22:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1248775168 bytes 2023-12-05T22:34:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1287528448 bytes 2023-12-05T22:34:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1306815488 bytes 2023-12-05T22:34:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1377679360 bytes 2023-12-05T22:34:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1421758464 bytes 2023-12-05T22:34:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1463856128 bytes 2023-12-05T22:34:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1494448128 bytes 2023-12-05T22:34:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1560758272 bytes 2023-12-05T22:34:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1594830848 bytes 2023-12-05T22:34:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1627137024 bytes 2023-12-05T22:34:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1681521664 bytes 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2023-12-05T22:34:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:34:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:34:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-05T22:34:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:34:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-05T22:34:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:34:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-05T22:34:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_12': 2023-12-05T22:34:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:34:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:34:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:34:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12f1a14cae6 2023-12-05T22:34:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12f1a14cae6 2023-12-05T22:34:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:34:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:34:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:34:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:34:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:18 2023 2023-12-05T22:34:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:18 2023 2023-12-05T22:34:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:18 2023 2023-12-05T22:34:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:34:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:34:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 73016fb1-14a6-445e-a3b2-f2065161bec3 2023-12-05T22:34:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:34:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:34:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-05T22:34:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:34:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:34:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:34:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:34:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:34:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 71593984 bytes 2023-12-05T22:34:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 101076992 bytes 2023-12-05T22:34:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 101961728 bytes 2023-12-05T22:34:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 130569216 bytes 2023-12-05T22:34:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 187921408 bytes 2023-12-05T22:34:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 242045952 bytes 2023-12-05T22:34:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 327415808 bytes 2023-12-05T22:34:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 365626368 bytes 2023-12-05T22:34:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 459442176 bytes 2023-12-05T22:34:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 521744384 bytes 2023-12-05T22:34:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 635316224 bytes 2023-12-05T22:34:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 661797888 bytes 2023-12-05T22:34:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 664942592 bytes 2023-12-05T22:34:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 733684736 bytes 2023-12-05T22:34:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 734285824 bytes 2023-12-05T22:34:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 784288768 bytes 2023-12-05T22:34:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 834785280 bytes 2023-12-05T22:34:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 872443904 bytes 2023-12-05T22:34:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 934908928 bytes 2023-12-05T22:34:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 980020224 bytes 2023-12-05T22:34:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1007577088 bytes 2023-12-05T22:34:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1026572288 bytes 2023-12-05T22:34:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1061188608 bytes 2023-12-05T22:34:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1106381824 bytes 2023-12-05T22:34:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1169687552 bytes 2023-12-05T22:34:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1218286592 bytes 2023-12-05T22:34:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1283922944 bytes 2023-12-05T22:34:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1325216768 bytes 2023-12-05T22:34:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1393608704 bytes 2023-12-05T22:34:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1434890240 bytes 2023-12-05T22:34:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1443450880 bytes 2023-12-05T22:34:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1506575360 bytes 2023-12-05T22:34:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1533111296 bytes 2023-12-05T22:34:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1576378368 bytes 2023-12-05T22:34:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1659887616 bytes 2023-12-05T22:34:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1693763584 bytes 2023-12-05T22:34:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1735901184 bytes 2023-12-05T22:34:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1750936576 bytes 2023-12-05T22:34:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1787233280 bytes 2023-12-05T22:34:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1827185664 bytes 2023-12-05T22:34:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1873669120 bytes 2023-12-05T22:34:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1893726208 bytes 2023-12-05T22:34:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1908614144 bytes 2023-12-05T22:34:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1931535360 bytes 2023-12-05T22:34:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2027259904 bytes 2023-12-05T22:34:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2063322112 bytes 2023-12-05T22:34:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2098414592 bytes 2023-12-05T22:34:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2138009600 bytes 2023-12-05T22:34:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2200417280 bytes 2023-12-05T22:34:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2275793920 bytes 2023-12-05T22:34:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2324389888 bytes 2023-12-05T22:34:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2355040256 bytes 2023-12-05T22:34:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2390809600 bytes 2023-12-05T22:34:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2406536192 bytes 2023-12-05T22:34:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2443395072 bytes 2023-12-05T22:34:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2484834304 bytes 2023-12-05T22:34:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2500473856 bytes 2023-12-05T22:34:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2623960064 bytes 2023-12-05T22:34:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2649308160 bytes 2023-12-05T22:34:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2671035392 bytes 2023-12-05T22:34:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2730635264 bytes 2023-12-05T22:34:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2756308992 bytes 2023-12-05T22:34:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2801529856 bytes 2023-12-05T22:34:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2806028288 bytes 2023-12-05T22:34:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2844306432 bytes 2023-12-05T22:34:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2844932096 bytes 2023-12-05T22:34:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2876065792 bytes 2023-12-05T22:34:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2886267904 bytes 2023-12-05T22:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2919127040 bytes 2023-12-05T22:34:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2964712448 bytes 2023-12-05T22:34:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2993612800 bytes 2023-12-05T22:34:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3025465344 bytes 2023-12-05T22:34:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3057427456 bytes 2023-12-05T22:34:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3133974528 bytes 2023-12-05T22:34:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3205277696 bytes 2023-12-05T22:34:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3250398208 bytes 2023-12-05T22:34:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3297423360 bytes 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:34:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:34:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:34:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-05T22:34:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:34:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_13': 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12fa336c9cee 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12fa336c9cee 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:34:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:34:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:33 2023 2023-12-05T22:34:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:33 2023 2023-12-05T22:34:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:33 2023 2023-12-05T22:34:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:34:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:34:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d5db6364-6a7e-48b6-a251-4e792c42f0d9 2023-12-05T22:34:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:34:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:34:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-05T22:34:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:34:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:34:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2023-12-05T22:34:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:34:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:34:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 27876352 bytes 2023-12-05T22:34:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 45824000 bytes 2023-12-05T22:34:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 50717696 bytes 2023-12-05T22:34:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 108275712 bytes 2023-12-05T22:34:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 137048064 bytes 2023-12-05T22:34:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 158271488 bytes 2023-12-05T22:34:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 245026816 bytes 2023-12-05T22:34:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 311090176 bytes 2023-12-05T22:34:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 363994112 bytes 2023-12-05T22:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 382023680 bytes 2023-12-05T22:34:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 420739072 bytes 2023-12-05T22:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 453464064 bytes 2023-12-05T22:34:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 477711360 bytes 2023-12-05T22:34:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 483732480 bytes 2023-12-05T22:34:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 492456960 bytes 2023-12-05T22:34:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 528126976 bytes 2023-12-05T22:34:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 616256512 bytes 2023-12-05T22:34:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 664070144 bytes 2023-12-05T22:34:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 684138496 bytes 2023-12-05T22:34:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 711027712 bytes 2023-12-05T22:34:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 759003136 bytes 2023-12-05T22:34:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 875195392 bytes 2023-12-05T22:34:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 910966784 bytes 2023-12-05T22:34:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 994409472 bytes 2023-12-05T22:34:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1051152384 bytes 2023-12-05T22:34:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1110670336 bytes 2023-12-05T22:34:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1129253888 bytes 2023-12-05T22:34:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1130798080 bytes 2023-12-05T22:34:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1149441024 bytes 2023-12-05T22:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-05T22:34:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:34:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-05T22:34:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_14': 2023-12-05T22:34:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:34:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13036bb04701 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13036bb04701 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:39 2023 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:39 2023 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:39 2023 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c9eb233a-812a-48e1-98fd-e7779e2fce61 2023-12-05T22:34:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:34:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:34:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-05T22:34:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:34:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2023-12-05T22:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:34:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 73637888 bytes 2023-12-05T22:34:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 83082240 bytes 2023-12-05T22:34:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 113087488 bytes 2023-12-05T22:34:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 187971584 bytes 2023-12-05T22:34:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 238585856 bytes 2023-12-05T22:34:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 263597056 bytes 2023-12-05T22:34:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 298459136 bytes 2023-12-05T22:34:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 326466560 bytes 2023-12-05T22:34:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 386995200 bytes 2023-12-05T22:34:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 505866240 bytes 2023-12-05T22:34:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 607514624 bytes 2023-12-05T22:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 620700672 bytes 2023-12-05T22:34:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 686608384 bytes 2023-12-05T22:34:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 755980288 bytes 2023-12-05T22:34:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 769827840 bytes 2023-12-05T22:34:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 806443008 bytes 2023-12-05T22:34:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 832607232 bytes 2023-12-05T22:34:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 854194176 bytes 2023-12-05T22:34:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 915529728 bytes 2023-12-05T22:34:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 958760960 bytes 2023-12-05T22:34:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1020360704 bytes 2023-12-05T22:34:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1091768320 bytes 2023-12-05T22:34:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1092370432 bytes 2023-12-05T22:34:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1113581568 bytes 2023-12-05T22:34:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1181414400 bytes 2023-12-05T22:34:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1207871488 bytes 2023-12-05T22:34:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1296718848 bytes 2023-12-05T22:34:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1333025792 bytes 2023-12-05T22:34:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1336315904 bytes 2023-12-05T22:34:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1386686464 bytes 2023-12-05T22:34:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1409135616 bytes 2023-12-05T22:34:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1512596480 bytes 2023-12-05T22:34:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1605523456 bytes 2023-12-05T22:34:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1606156288 bytes 2023-12-05T22:34:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1647890432 bytes 2023-12-05T22:34:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1741689856 bytes 2023-12-05T22:34:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1744584704 bytes 2023-12-05T22:34:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1840349184 bytes 2023-12-05T22:34:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1930009600 bytes 2023-12-05T22:34:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1970079744 bytes 2023-12-05T22:34:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2012468224 bytes 2023-12-05T22:34:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2065885184 bytes 2023-12-05T22:34:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2087561216 bytes 2023-12-05T22:34:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2122893312 bytes 2023-12-05T22:34:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2153658368 bytes 2023-12-05T22:34:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2185922560 bytes 2023-12-05T22:34:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2262591488 bytes 2023-12-05T22:34:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2283850752 bytes 2023-12-05T22:34:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2351457280 bytes 2023-12-05T22:34:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2410238976 bytes 2023-12-05T22:34:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2426861568 bytes 2023-12-05T22:34:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2497491968 bytes 2023-12-05T22:34:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2540182528 bytes 2023-12-05T22:34:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2561454080 bytes 2023-12-05T22:34:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2603681792 bytes 2023-12-05T22:34:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2641192960 bytes 2023-12-05T22:34:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2734066688 bytes 2023-12-05T22:34:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2782330880 bytes 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:34:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:34:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2023-12-05T22:34:51.342 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-05T22:34:51.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-05T22:34:51.343 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-05T22:34:51.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:34:51.351+0000 7f923fe99640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 46501) UID: 1000 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-05T22:34:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:34:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_15': 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 130e30aa0a15 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.130e30aa0a15 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:34:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:51 2023 2023-12-05T22:34:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:51 2023 2023-12-05T22:34:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:51 2023 2023-12-05T22:34:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:34:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:34:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b9b1e588-b890-4a39-a107-1d78081756ec 2023-12-05T22:34:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-05T22:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:34:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2023-12-05T22:34:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:34:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:34:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 68927488 bytes 2023-12-05T22:34:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 76368896 bytes 2023-12-05T22:34:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 155958272 bytes 2023-12-05T22:34:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 216619008 bytes 2023-12-05T22:34:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 228992000 bytes 2023-12-05T22:34:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 229874688 bytes 2023-12-05T22:34:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 269626368 bytes 2023-12-05T22:34:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 358787072 bytes 2023-12-05T22:34:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 428702720 bytes 2023-12-05T22:34:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 519555072 bytes 2023-12-05T22:34:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 566361088 bytes 2023-12-05T22:34:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 606087168 bytes 2023-12-05T22:34:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 646861824 bytes 2023-12-05T22:34:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 697961472 bytes 2023-12-05T22:34:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 720036864 bytes 2023-12-05T22:34:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 785334272 bytes 2023-12-05T22:34:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 842971136 bytes 2023-12-05T22:34:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 863890432 bytes 2023-12-05T22:34:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 914521088 bytes 2023-12-05T22:34:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 957751296 bytes 2023-12-05T22:34:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 989306880 bytes 2023-12-05T22:34:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1023273984 bytes 2023-12-05T22:34:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1049476096 bytes 2023-12-05T22:34:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1105519616 bytes 2023-12-05T22:34:56.344 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-05T22:34:56.344 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:34:56.345 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-05T22:34:56.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-05T22:34:56.345 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-05T22:34:56.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-05T22:34:56.348 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-05T22:34:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1135243264 bytes 2023-12-05T22:34:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1148730368 bytes 2023-12-05T22:34:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1202827264 bytes 2023-12-05T22:34:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1245410304 bytes 2023-12-05T22:34:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1259531264 bytes 2023-12-05T22:34:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1328677888 bytes 2023-12-05T22:34:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1395510272 bytes 2023-12-05T22:34:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:34:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-05T22:34:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:34:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:34:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-05T22:34:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:34:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_16': 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 131d78ac145f 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.131d78ac145f 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:57 2023 2023-12-05T22:34:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:57 2023 2023-12-05T22:34:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:57 2023 2023-12-05T22:34:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:34:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:34:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b3ff1efc-2dee-4937-993c-fb819906b94f 2023-12-05T22:34:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:34:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:34:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-05T22:34:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:34:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:34:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:34:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:34:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2023-12-05T22:34:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:34:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:34:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 34167808 bytes 2023-12-05T22:34:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 70976512 bytes 2023-12-05T22:34:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 74014720 bytes 2023-12-05T22:34:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 100784128 bytes 2023-12-05T22:34:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 139971584 bytes 2023-12-05T22:34:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 178515968 bytes 2023-12-05T22:34:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 227238912 bytes 2023-12-05T22:34:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 247922688 bytes 2023-12-05T22:34:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 321630208 bytes 2023-12-05T22:34:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 345394176 bytes 2023-12-05T22:35:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 380538880 bytes 2023-12-05T22:35:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 395307008 bytes 2023-12-05T22:35:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 469416960 bytes 2023-12-05T22:35:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 520725504 bytes 2023-12-05T22:35:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 537093120 bytes 2023-12-05T22:35:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 642664448 bytes 2023-12-05T22:35:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 675077120 bytes 2023-12-05T22:35:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 735504384 bytes 2023-12-05T22:35:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 736129024 bytes 2023-12-05T22:35:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 777791488 bytes 2023-12-05T22:35:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 821158912 bytes 2023-12-05T22:35:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 845888512 bytes 2023-12-05T22:35:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 857230336 bytes 2023-12-05T22:35:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 887980032 bytes 2023-12-05T22:35:02.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 910453760 bytes 2023-12-05T22:35:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 917554176 bytes 2023-12-05T22:35:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1015981056 bytes 2023-12-05T22:35:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1086921728 bytes 2023-12-05T22:35:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1124765696 bytes 2023-12-05T22:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1160612864 bytes 2023-12-05T22:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1228123136 bytes 2023-12-05T22:35:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1260903424 bytes 2023-12-05T22:35:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1323867136 bytes 2023-12-05T22:35:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1384887296 bytes 2023-12-05T22:35:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1421731840 bytes 2023-12-05T22:35:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1486409728 bytes 2023-12-05T22:35:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1487029248 bytes 2023-12-05T22:35:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1524677632 bytes 2023-12-05T22:35:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1597805568 bytes 2023-12-05T22:35:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1660510208 bytes 2023-12-05T22:35:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1699390464 bytes 2023-12-05T22:35:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1754454016 bytes 2023-12-05T22:35:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1793373184 bytes 2023-12-05T22:35:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1840263168 bytes 2023-12-05T22:35:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:35:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:35:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:35:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2023-12-05T22:35:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:35:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:35:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:35:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:35:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2023-12-05T22:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-05T22:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-05T22:35:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:35:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-05T22:35:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_17': 2023-12-05T22:35:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:35:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1328738d0ddb 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1328738d0ddb 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:06 2023 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:06 2023 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:06 2023 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:35:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9f56dd88-b64c-4071-b253-84119be63f20 2023-12-05T22:35:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-05T22:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:35:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:35:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:35:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:35:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 21105664 bytes 2023-12-05T22:35:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 100029440 bytes 2023-12-05T22:35:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 131364864 bytes 2023-12-05T22:35:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 189608960 bytes 2023-12-05T22:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 245658624 bytes 2023-12-05T22:35:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 257932288 bytes 2023-12-05T22:35:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 345010176 bytes 2023-12-05T22:35:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 442567680 bytes 2023-12-05T22:35:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 476721152 bytes 2023-12-05T22:35:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 503198720 bytes 2023-12-05T22:35:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 563561472 bytes 2023-12-05T22:35:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 564174848 bytes 2023-12-05T22:35:09.813 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:35:09.810+0000 7f1618813640 -1 reset not still connected to 0x558e24c01930 2023-12-05T22:35:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 662947840 bytes 2023-12-05T22:35:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 663569408 bytes 2023-12-05T22:35:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 706553856 bytes 2023-12-05T22:35:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 729932800 bytes 2023-12-05T22:35:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 769045504 bytes 2023-12-05T22:35:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 800368640 bytes 2023-12-05T22:35:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 893301760 bytes 2023-12-05T22:35:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 910146560 bytes 2023-12-05T22:35:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 944103424 bytes 2023-12-05T22:35:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 970974208 bytes 2023-12-05T22:35:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1019030528 bytes 2023-12-05T22:35:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1183247360 bytes 2023-12-05T22:35:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1233230848 bytes 2023-12-05T22:35:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1294428160 bytes 2023-12-05T22:35:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1351486464 bytes 2023-12-05T22:35:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1383301120 bytes 2023-12-05T22:35:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1472638976 bytes 2023-12-05T22:35:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1532067840 bytes 2023-12-05T22:35:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1564718080 bytes 2023-12-05T22:35:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1588590592 bytes 2023-12-05T22:35:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1626487808 bytes 2023-12-05T22:35:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1689630720 bytes 2023-12-05T22:35:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1720601600 bytes 2023-12-05T22:35:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1767971840 bytes 2023-12-05T22:35:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1799196672 bytes 2023-12-05T22:35:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1817600000 bytes 2023-12-05T22:35:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1843402752 bytes 2023-12-05T22:35:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1900286976 bytes 2023-12-05T22:35:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1907488768 bytes 2023-12-05T22:35:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1912482816 bytes 2023-12-05T22:35:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1962680320 bytes 2023-12-05T22:35:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1981302784 bytes 2023-12-05T22:35:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2031097856 bytes 2023-12-05T22:35:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2101116928 bytes 2023-12-05T22:35:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2119506944 bytes 2023-12-05T22:35:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2180588544 bytes 2023-12-05T22:35:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2199223296 bytes 2023-12-05T22:35:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2238452736 bytes 2023-12-05T22:35:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2286044160 bytes 2023-12-05T22:35:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2324678656 bytes 2023-12-05T22:35:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2352259072 bytes 2023-12-05T22:35:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2369059840 bytes 2023-12-05T22:35:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2421345280 bytes 2023-12-05T22:35:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2471348224 bytes 2023-12-05T22:35:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2546600960 bytes 2023-12-05T22:35:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2578151424 bytes 2023-12-05T22:35:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2619753472 bytes 2023-12-05T22:35:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2705594368 bytes 2023-12-05T22:35:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2722107392 bytes 2023-12-05T22:35:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2746612736 bytes 2023-12-05T22:35:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2833235968 bytes 2023-12-05T22:35:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2873872384 bytes 2023-12-05T22:35:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2901089280 bytes 2023-12-05T22:35:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2946997248 bytes 2023-12-05T22:35:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2987460608 bytes 2023-12-05T22:35:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3004611584 bytes 2023-12-05T22:35:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3071080448 bytes 2023-12-05T22:35:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:35:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:35:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:35:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:35:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:35:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:35:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:35:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2023-12-05T22:35:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:35:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:35:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-05T22:35:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:35:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-05T22:35:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:35:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_18': 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 133078263668 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.133078263668 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:20 2023 2023-12-05T22:35:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:20 2023 2023-12-05T22:35:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:20 2023 2023-12-05T22:35:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:35:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:35:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: a058c447-ff5e-49d0-9009-af35f59afbb5 2023-12-05T22:35:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:35:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:35:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-05T22:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2023-12-05T22:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:35:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 10118144 bytes 2023-12-05T22:35:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 20583424 bytes 2023-12-05T22:35:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 92775424 bytes 2023-12-05T22:35:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 170761216 bytes 2023-12-05T22:35:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 189950976 bytes 2023-12-05T22:35:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 241731584 bytes 2023-12-05T22:35:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 386200576 bytes 2023-12-05T22:35:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 392250368 bytes 2023-12-05T22:35:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 405072896 bytes 2023-12-05T22:35:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 414559232 bytes 2023-12-05T22:35:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 462021632 bytes 2023-12-05T22:35:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 507975680 bytes 2023-12-05T22:35:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 513016832 bytes 2023-12-05T22:35:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 621568000 bytes 2023-12-05T22:35:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 639667200 bytes 2023-12-05T22:35:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 688976896 bytes 2023-12-05T22:35:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 695040000 bytes 2023-12-05T22:35:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 706597888 bytes 2023-12-05T22:35:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 805184512 bytes 2023-12-05T22:35:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 849549312 bytes 2023-12-05T22:35:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 852382720 bytes 2023-12-05T22:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 935568384 bytes 2023-12-05T22:35:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1008311296 bytes 2023-12-05T22:35:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1033723904 bytes 2023-12-05T22:35:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1092948992 bytes 2023-12-05T22:35:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1112515584 bytes 2023-12-05T22:35:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1141291008 bytes 2023-12-05T22:35:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:35:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:35:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:35:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:35:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:35:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:35:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:35:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2023-12-05T22:35:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:35:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:35:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-05T22:35:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:35:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-05T22:35:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:35:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-05T22:35:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_19': 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1339abc86968 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1339abc86968 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:26 2023 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:26 2023 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:26 2023 2023-12-05T22:35:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:35:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:35:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e72c6cbf-3629-4ab9-855f-c2cdd06d81a7 2023-12-05T22:35:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:35:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:35:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-05T22:35:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:35:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:35:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:35:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:35:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:35:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 39426048 bytes 2023-12-05T22:35:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 84226048 bytes 2023-12-05T22:35:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 170985472 bytes 2023-12-05T22:35:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 247956480 bytes 2023-12-05T22:35:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 297435136 bytes 2023-12-05T22:35:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 319587328 bytes 2023-12-05T22:35:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 332733440 bytes 2023-12-05T22:35:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 364776448 bytes 2023-12-05T22:35:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 419769344 bytes 2023-12-05T22:35:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 437953536 bytes 2023-12-05T22:35:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 460603392 bytes 2023-12-05T22:35:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 506640384 bytes 2023-12-05T22:35:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 526060544 bytes 2023-12-05T22:35:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 547375104 bytes 2023-12-05T22:35:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 585916416 bytes 2023-12-05T22:35:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 610332672 bytes 2023-12-05T22:35:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 610964480 bytes 2023-12-05T22:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 632386560 bytes 2023-12-05T22:35:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 704667648 bytes 2023-12-05T22:35:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 744614912 bytes 2023-12-05T22:35:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 776796160 bytes 2023-12-05T22:35:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 888755200 bytes 2023-12-05T22:35:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1011622912 bytes 2023-12-05T22:35:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1051702272 bytes 2023-12-05T22:35:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1110118400 bytes 2023-12-05T22:35:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1139775488 bytes 2023-12-05T22:35:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1178669056 bytes 2023-12-05T22:35:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1254031360 bytes 2023-12-05T22:35:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1351153664 bytes 2023-12-05T22:35:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1414132736 bytes 2023-12-05T22:35:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1464210432 bytes 2023-12-05T22:35:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1540107264 bytes 2023-12-05T22:35:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1611505664 bytes 2023-12-05T22:35:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1651832832 bytes 2023-12-05T22:35:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1700411392 bytes 2023-12-05T22:35:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1784586240 bytes 2023-12-05T22:35:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1826149376 bytes 2023-12-05T22:35:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1876506624 bytes 2023-12-05T22:35:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1892471808 bytes 2023-12-05T22:35:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1936753664 bytes 2023-12-05T22:35:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1989646336 bytes 2023-12-05T22:35:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2001095680 bytes 2023-12-05T22:35:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2044231680 bytes 2023-12-05T22:35:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2128643072 bytes 2023-12-05T22:35:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2129272832 bytes 2023-12-05T22:35:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2143897600 bytes 2023-12-05T22:35:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2189563904 bytes 2023-12-05T22:35:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2245110784 bytes 2023-12-05T22:35:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2269282304 bytes 2023-12-05T22:35:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2350688256 bytes 2023-12-05T22:35:37.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2366370816 bytes 2023-12-05T22:35:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2408850432 bytes 2023-12-05T22:35:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2462952448 bytes 2023-12-05T22:35:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2589010944 bytes 2023-12-05T22:35:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2615105536 bytes 2023-12-05T22:35:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2711199744 bytes 2023-12-05T22:35:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2715538432 bytes 2023-12-05T22:35:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2818363392 bytes 2023-12-05T22:35:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2854425600 bytes 2023-12-05T22:35:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2874831872 bytes 2023-12-05T22:35:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2920269824 bytes 2023-12-05T22:35:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2977821696 bytes 2023-12-05T22:35:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 3003305984 bytes 2023-12-05T22:35:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 3040736256 bytes 2023-12-05T22:35:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3095331840 bytes 2023-12-05T22:35:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3213231104 bytes 2023-12-05T22:35:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3235198976 bytes 2023-12-05T22:35:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3255417856 bytes 2023-12-05T22:35:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3294820352 bytes 2023-12-05T22:35:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3360990208 bytes 2023-12-05T22:35:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:35:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:35:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:35:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:35:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2023-12-05T22:35:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-05T22:35:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:35:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-05T22:35:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_20': 2023-12-05T22:35:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:35:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:35:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:35:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1343ad0f96cf 2023-12-05T22:35:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1343ad0f96cf 2023-12-05T22:35:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:41 2023 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:41 2023 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:41 2023 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1533eb9c-c4ac-468b-9060-8f72f4f33772 2023-12-05T22:35:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:35:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:35:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-05T22:35:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:35:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:35:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2023-12-05T22:35:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:35:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:35:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 38526976 bytes 2023-12-05T22:35:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 52530176 bytes 2023-12-05T22:35:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 120324096 bytes 2023-12-05T22:35:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 165624832 bytes 2023-12-05T22:35:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 230157312 bytes 2023-12-05T22:35:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 260355072 bytes 2023-12-05T22:35:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 267290624 bytes 2023-12-05T22:35:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 315583488 bytes 2023-12-05T22:35:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 389866496 bytes 2023-12-05T22:35:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 410701824 bytes 2023-12-05T22:35:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 425167872 bytes 2023-12-05T22:35:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 440895488 bytes 2023-12-05T22:35:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 488878080 bytes 2023-12-05T22:35:44.344 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-05T22:35:44.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-05T22:35:44.345 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-05T22:35:44.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:35:44.354+0000 7f2978efb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48654) UID: 1000 2023-12-05T22:35:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 554971136 bytes 2023-12-05T22:35:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 562902016 bytes 2023-12-05T22:35:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 576459776 bytes 2023-12-05T22:35:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 623124480 bytes 2023-12-05T22:35:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 661135360 bytes 2023-12-05T22:35:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 717541376 bytes 2023-12-05T22:35:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 766750720 bytes 2023-12-05T22:35:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 790042624 bytes 2023-12-05T22:35:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 815096832 bytes 2023-12-05T22:35:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 849252352 bytes 2023-12-05T22:35:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 856010752 bytes 2023-12-05T22:35:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 907664384 bytes 2023-12-05T22:35:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 919234560 bytes 2023-12-05T22:35:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 941804544 bytes 2023-12-05T22:35:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 986920960 bytes 2023-12-05T22:35:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1053670400 bytes 2023-12-05T22:35:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1092954112 bytes 2023-12-05T22:35:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1158834176 bytes 2023-12-05T22:35:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1195506688 bytes 2023-12-05T22:35:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1249213440 bytes 2023-12-05T22:35:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1289749504 bytes 2023-12-05T22:35:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1343620096 bytes 2023-12-05T22:35:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1410521088 bytes 2023-12-05T22:35:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1421931520 bytes 2023-12-05T22:35:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1482384384 bytes 2023-12-05T22:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1550921728 bytes 2023-12-05T22:35:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1588628480 bytes 2023-12-05T22:35:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1610894336 bytes 2023-12-05T22:35:49.346 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-05T22:35:49.346 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:35:49.346 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-05T22:35:49.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-05T22:35:49.347 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-05T22:35:49.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-05T22:35:49.349 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-05T22:35:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1698125824 bytes 2023-12-05T22:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1724029952 bytes 2023-12-05T22:35:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1764319232 bytes 2023-12-05T22:35:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1831836672 bytes 2023-12-05T22:35:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1944281088 bytes 2023-12-05T22:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1977060352 bytes 2023-12-05T22:35:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2053941248 bytes 2023-12-05T22:35:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2090169344 bytes 2023-12-05T22:35:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2092290048 bytes 2023-12-05T22:35:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2128712704 bytes 2023-12-05T22:35:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2143267840 bytes 2023-12-05T22:35:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2165713920 bytes 2023-12-05T22:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2201684992 bytes 2023-12-05T22:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2023-12-05T22:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:35:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:35:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:35:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:35:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:35:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2023-12-05T22:35:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:35:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:35:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:35:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-05T22:35:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:35:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:35:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-05T22:35:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:35:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-05T22:35:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_21': 2023-12-05T22:35:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:35:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:35:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:35:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 135398237eaa 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.135398237eaa 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:51 2023 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:51 2023 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:51 2023 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 18f502e6-d626-414b-b187-fc0882719611 2023-12-05T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-05T22:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:35:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:35:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:35:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:35:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 25781248 bytes 2023-12-05T22:35:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 79182848 bytes 2023-12-05T22:35:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 87398400 bytes 2023-12-05T22:35:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 135030784 bytes 2023-12-05T22:35:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 199558144 bytes 2023-12-05T22:35:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 277918720 bytes 2023-12-05T22:35:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 278528000 bytes 2023-12-05T22:35:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 336395264 bytes 2023-12-05T22:35:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 406433792 bytes 2023-12-05T22:35:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 411001856 bytes 2023-12-05T22:35:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 457592832 bytes 2023-12-05T22:35:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 476827648 bytes 2023-12-05T22:35:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 545500160 bytes 2023-12-05T22:35:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 576720896 bytes 2023-12-05T22:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 646743040 bytes 2023-12-05T22:35:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 706682880 bytes 2023-12-05T22:35:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 776465408 bytes 2023-12-05T22:35:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 792301568 bytes 2023-12-05T22:35:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 849385472 bytes 2023-12-05T22:35:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 915766272 bytes 2023-12-05T22:35:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 999288832 bytes 2023-12-05T22:35:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1067954176 bytes 2023-12-05T22:35:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1109776384 bytes 2023-12-05T22:35:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1156936704 bytes 2023-12-05T22:35:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1157555200 bytes 2023-12-05T22:35:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1202839552 bytes 2023-12-05T22:35:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1203454976 bytes 2023-12-05T22:35:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1218707456 bytes 2023-12-05T22:35:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1255716864 bytes 2023-12-05T22:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1336222720 bytes 2023-12-05T22:35:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1410588672 bytes 2023-12-05T22:35:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1475793920 bytes 2023-12-05T22:35:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1501254656 bytes 2023-12-05T22:35:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1531111424 bytes 2023-12-05T22:35:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1571347456 bytes 2023-12-05T22:35:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1617780736 bytes 2023-12-05T22:35:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1645979648 bytes 2023-12-05T22:36:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1714539520 bytes 2023-12-05T22:36:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1762398208 bytes 2023-12-05T22:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1775724544 bytes 2023-12-05T22:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1807703040 bytes 2023-12-05T22:36:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1816718336 bytes 2023-12-05T22:36:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1851018240 bytes 2023-12-05T22:36:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1869250560 bytes 2023-12-05T22:36:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1927234560 bytes 2023-12-05T22:36:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2024772608 bytes 2023-12-05T22:36:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2082996224 bytes 2023-12-05T22:36:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2158917632 bytes 2023-12-05T22:36:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2210584576 bytes 2023-12-05T22:36:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2214637568 bytes 2023-12-05T22:36:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2279836672 bytes 2023-12-05T22:36:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2301314048 bytes 2023-12-05T22:36:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2347206656 bytes 2023-12-05T22:36:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2395676672 bytes 2023-12-05T22:36:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2429147136 bytes 2023-12-05T22:36:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2473405440 bytes 2023-12-05T22:36:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2503795712 bytes 2023-12-05T22:36:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2509535232 bytes 2023-12-05T22:36:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2565234688 bytes 2023-12-05T22:36:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2637794304 bytes 2023-12-05T22:36:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2638409728 bytes 2023-12-05T22:36:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2698673152 bytes 2023-12-05T22:36:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2719686656 bytes 2023-12-05T22:36:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2757008384 bytes 2023-12-05T22:36:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2023-12-05T22:36:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:36:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-05T22:36:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:36:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-05T22:36:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_22': 2023-12-05T22:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 135d1b92159a 2023-12-05T22:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.135d1b92159a 2023-12-05T22:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:04 2023 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:04 2023 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:04 2023 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:36:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e296de3e-e023-46cc-9b13-df738af23268 2023-12-05T22:36:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:36:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:36:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-05T22:36:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:36:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:36:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2023-12-05T22:36:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:36:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:36:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 616448 bytes 2023-12-05T22:36:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 65714176 bytes 2023-12-05T22:36:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 89303040 bytes 2023-12-05T22:36:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 147388416 bytes 2023-12-05T22:36:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 189332480 bytes 2023-12-05T22:36:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 262854656 bytes 2023-12-05T22:36:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 317825024 bytes 2023-12-05T22:36:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 394196992 bytes 2023-12-05T22:36:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 419859456 bytes 2023-12-05T22:36:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 444386304 bytes 2023-12-05T22:36:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 444993536 bytes 2023-12-05T22:36:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 495959040 bytes 2023-12-05T22:36:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 509689856 bytes 2023-12-05T22:36:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 523712512 bytes 2023-12-05T22:36:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 591879168 bytes 2023-12-05T22:36:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 622535680 bytes 2023-12-05T22:36:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 671757312 bytes 2023-12-05T22:36:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 692615168 bytes 2023-12-05T22:36:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 722811904 bytes 2023-12-05T22:36:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 747725824 bytes 2023-12-05T22:36:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 783286272 bytes 2023-12-05T22:36:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 825852928 bytes 2023-12-05T22:36:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 912929792 bytes 2023-12-05T22:36:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 954746880 bytes 2023-12-05T22:36:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 961044480 bytes 2023-12-05T22:36:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1020592128 bytes 2023-12-05T22:36:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1059123200 bytes 2023-12-05T22:36:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:36:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:36:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:36:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2023-12-05T22:36:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:36:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:36:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:36:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:36:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:36:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-05T22:36:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:36:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-05T22:36:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_23': 2023-12-05T22:36:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1366102b92ef 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1366102b92ef 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:10 2023 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:10 2023 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:10 2023 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:36:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e9b3d2ae-1928-4b14-803b-779daf73fd91 2023-12-05T22:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:36:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-05T22:36:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:36:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:36:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2023-12-05T22:36:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:36:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:36:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 49810432 bytes 2023-12-05T22:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 137984000 bytes 2023-12-05T22:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 138621952 bytes 2023-12-05T22:36:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 155753472 bytes 2023-12-05T22:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 197056512 bytes 2023-12-05T22:36:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 250706944 bytes 2023-12-05T22:36:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 274091008 bytes 2023-12-05T22:36:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 279327744 bytes 2023-12-05T22:36:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 357432320 bytes 2023-12-05T22:36:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 358059008 bytes 2023-12-05T22:36:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 358704128 bytes 2023-12-05T22:36:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 384091136 bytes 2023-12-05T22:36:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 400552960 bytes 2023-12-05T22:36:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 485078016 bytes 2023-12-05T22:36:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 510042112 bytes 2023-12-05T22:36:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 594379776 bytes 2023-12-05T22:36:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 614665216 bytes 2023-12-05T22:36:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 625580032 bytes 2023-12-05T22:36:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 677483520 bytes 2023-12-05T22:36:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 684341248 bytes 2023-12-05T22:36:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 696855552 bytes 2023-12-05T22:36:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 745004032 bytes 2023-12-05T22:36:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 831686656 bytes 2023-12-05T22:36:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 864267264 bytes 2023-12-05T22:36:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 907735040 bytes 2023-12-05T22:36:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 969230336 bytes 2023-12-05T22:36:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 986477568 bytes 2023-12-05T22:36:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1036874752 bytes 2023-12-05T22:36:16.260 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:36:16.269+0000 7fdd570d4640 -1 reset not still connected to 0x559d304184e0 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:36:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:36:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:36:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2023-12-05T22:36:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-05T22:36:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:36:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_24': 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 136eafab7918 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.136eafab7918 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:36:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:16 2023 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:16 2023 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:16 2023 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 17202bda-24c3-40df-b59d-9904509b75df 2023-12-05T22:36:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:36:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:36:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-05T22:36:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:36:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:36:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2023-12-05T22:36:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:36:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:36:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 20378624 bytes 2023-12-05T22:36:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 46154752 bytes 2023-12-05T22:36:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 70005760 bytes 2023-12-05T22:36:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 75163648 bytes 2023-12-05T22:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 156385280 bytes 2023-12-05T22:36:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 191081472 bytes 2023-12-05T22:36:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 209496064 bytes 2023-12-05T22:36:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 286254080 bytes 2023-12-05T22:36:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 330716160 bytes 2023-12-05T22:36:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 391358464 bytes 2023-12-05T22:36:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 453109760 bytes 2023-12-05T22:36:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 492476416 bytes 2023-12-05T22:36:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 512400384 bytes 2023-12-05T22:36:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 559306752 bytes 2023-12-05T22:36:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 600217600 bytes 2023-12-05T22:36:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 625146880 bytes 2023-12-05T22:36:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 667447296 bytes 2023-12-05T22:36:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 668047360 bytes 2023-12-05T22:36:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 758786048 bytes 2023-12-05T22:36:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 819917824 bytes 2023-12-05T22:36:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 867845120 bytes 2023-12-05T22:36:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 933524480 bytes 2023-12-05T22:36:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 968189952 bytes 2023-12-05T22:36:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 993876992 bytes 2023-12-05T22:36:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1102318592 bytes 2023-12-05T22:36:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1140743168 bytes 2023-12-05T22:36:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1194343424 bytes 2023-12-05T22:36:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1203217408 bytes 2023-12-05T22:36:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1253712896 bytes 2023-12-05T22:36:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1275131904 bytes 2023-12-05T22:36:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1301310464 bytes 2023-12-05T22:36:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1361234944 bytes 2023-12-05T22:36:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1388322816 bytes 2023-12-05T22:36:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1418660864 bytes 2023-12-05T22:36:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1487602688 bytes 2023-12-05T22:36:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1501426688 bytes 2023-12-05T22:36:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1515973632 bytes 2023-12-05T22:36:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1545467904 bytes 2023-12-05T22:36:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1596437504 bytes 2023-12-05T22:36:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1629093888 bytes 2023-12-05T22:36:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1629717504 bytes 2023-12-05T22:36:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1680643072 bytes 2023-12-05T22:36:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1738492928 bytes 2023-12-05T22:36:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1739118592 bytes 2023-12-05T22:36:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1797347328 bytes 2023-12-05T22:36:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1833454592 bytes 2023-12-05T22:36:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1848580096 bytes 2023-12-05T22:36:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1850744832 bytes 2023-12-05T22:36:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1895107584 bytes 2023-12-05T22:36:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:36:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:36:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2023-12-05T22:36:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:36:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-05T22:36:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:36:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-05T22:36:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_25': 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1376554af93 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1376554af93 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:26 2023 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:26 2023 2023-12-05T22:36:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:26 2023 2023-12-05T22:36:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:36:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:36:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4ff38d91-f124-4824-8d8b-0b2a3eb228a9 2023-12-05T22:36:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:36:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:36:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-05T22:36:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:36:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:36:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2023-12-05T22:36:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:36:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:36:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 20252672 bytes 2023-12-05T22:36:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 61729792 bytes 2023-12-05T22:36:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 122914816 bytes 2023-12-05T22:36:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 169534464 bytes 2023-12-05T22:36:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 212453376 bytes 2023-12-05T22:36:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 334351360 bytes 2023-12-05T22:36:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 371874816 bytes 2023-12-05T22:36:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 464987136 bytes 2023-12-05T22:36:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 498575360 bytes 2023-12-05T22:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 551116800 bytes 2023-12-05T22:36:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 570766336 bytes 2023-12-05T22:36:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 571376640 bytes 2023-12-05T22:36:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 593639424 bytes 2023-12-05T22:36:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 614866944 bytes 2023-12-05T22:36:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 655012864 bytes 2023-12-05T22:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 694343680 bytes 2023-12-05T22:36:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 733414400 bytes 2023-12-05T22:36:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 785729536 bytes 2023-12-05T22:36:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 817976320 bytes 2023-12-05T22:36:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 899144704 bytes 2023-12-05T22:36:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 936784896 bytes 2023-12-05T22:36:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 972485632 bytes 2023-12-05T22:36:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1003141120 bytes 2023-12-05T22:36:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1029412864 bytes 2023-12-05T22:36:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1046632448 bytes 2023-12-05T22:36:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1065149440 bytes 2023-12-05T22:36:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1126404096 bytes 2023-12-05T22:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1167687680 bytes 2023-12-05T22:36:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1190297600 bytes 2023-12-05T22:36:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1215385600 bytes 2023-12-05T22:36:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1236729856 bytes 2023-12-05T22:36:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1275857920 bytes 2023-12-05T22:36:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1288029184 bytes 2023-12-05T22:36:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1322483712 bytes 2023-12-05T22:36:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1348444160 bytes 2023-12-05T22:36:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1373336576 bytes 2023-12-05T22:36:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1441858560 bytes 2023-12-05T22:36:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1499297792 bytes 2023-12-05T22:36:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1518976000 bytes 2023-12-05T22:36:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1541556224 bytes 2023-12-05T22:36:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1616759808 bytes 2023-12-05T22:36:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1691629568 bytes 2023-12-05T22:36:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1698715648 bytes 2023-12-05T22:36:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1748255744 bytes 2023-12-05T22:36:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1855142912 bytes 2023-12-05T22:36:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:36:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:36:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:36:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-05T22:36:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2023-12-05T22:36:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:36:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:36:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-05T22:36:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:36:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-05T22:36:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:36:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_26': 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13808c9a4ab0 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13808c9a4ab0 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:36 2023 2023-12-05T22:36:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:36 2023 2023-12-05T22:36:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:36 2023 2023-12-05T22:36:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:36:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:36:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 119a68f7-e780-449a-acf3-897b2c081275 2023-12-05T22:36:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:36:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:36:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-05T22:36:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:36:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:36:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:36:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:36:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:36:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 24417280 bytes 2023-12-05T22:36:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 52207616 bytes 2023-12-05T22:36:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 63493120 bytes 2023-12-05T22:36:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 133473280 bytes 2023-12-05T22:36:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 161383424 bytes 2023-12-05T22:36:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 209556480 bytes 2023-12-05T22:36:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 230585344 bytes 2023-12-05T22:36:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 302623744 bytes 2023-12-05T22:36:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 335640576 bytes 2023-12-05T22:36:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 350891008 bytes 2023-12-05T22:36:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 428229632 bytes 2023-12-05T22:36:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 468363264 bytes 2023-12-05T22:36:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 480692224 bytes 2023-12-05T22:36:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 520951808 bytes 2023-12-05T22:36:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 578674688 bytes 2023-12-05T22:36:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 625497088 bytes 2023-12-05T22:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 626104320 bytes 2023-12-05T22:36:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 673206272 bytes 2023-12-05T22:36:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 761833472 bytes 2023-12-05T22:36:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 794555392 bytes 2023-12-05T22:36:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 830204928 bytes 2023-12-05T22:36:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 830812160 bytes 2023-12-05T22:36:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 864921600 bytes 2023-12-05T22:36:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 871711744 bytes 2023-12-05T22:36:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 924445696 bytes 2023-12-05T22:36:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 982398976 bytes 2023-12-05T22:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1007304704 bytes 2023-12-05T22:36:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1037077504 bytes 2023-12-05T22:36:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1078729728 bytes 2023-12-05T22:36:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1087099904 bytes 2023-12-05T22:36:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1100042240 bytes 2023-12-05T22:36:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1196639232 bytes 2023-12-05T22:36:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1265745920 bytes 2023-12-05T22:36:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1365913600 bytes 2023-12-05T22:36:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1416853504 bytes 2023-12-05T22:36:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1458313216 bytes 2023-12-05T22:36:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1458953216 bytes 2023-12-05T22:36:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1490155520 bytes 2023-12-05T22:36:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1512396800 bytes 2023-12-05T22:36:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1584451584 bytes 2023-12-05T22:36:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1654024192 bytes 2023-12-05T22:36:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1665950720 bytes 2023-12-05T22:36:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1740268544 bytes 2023-12-05T22:36:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1741353984 bytes 2023-12-05T22:36:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1760495616 bytes 2023-12-05T22:36:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1799533568 bytes 2023-12-05T22:36:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1832331264 bytes 2023-12-05T22:36:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1855507456 bytes 2023-12-05T22:36:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1865950208 bytes 2023-12-05T22:36:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1894540288 bytes 2023-12-05T22:36:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1974477824 bytes 2023-12-05T22:36:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2040688640 bytes 2023-12-05T22:36:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2085995520 bytes 2023-12-05T22:36:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2113417216 bytes 2023-12-05T22:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2131934208 bytes 2023-12-05T22:36:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2197591040 bytes 2023-12-05T22:36:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2295306240 bytes 2023-12-05T22:36:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2386329600 bytes 2023-12-05T22:36:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2427365376 bytes 2023-12-05T22:36:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2459813888 bytes 2023-12-05T22:36:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2492571648 bytes 2023-12-05T22:36:48.736 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:36:48.744+0000 7fdd570d4640 -1 reset not still connected to 0x559d267c3520 2023-12-05T22:36:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2546736128 bytes 2023-12-05T22:36:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2547350528 bytes 2023-12-05T22:36:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2602014720 bytes 2023-12-05T22:36:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2626917376 bytes 2023-12-05T22:36:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2643324928 bytes 2023-12-05T22:36:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2683871232 bytes 2023-12-05T22:36:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2705923072 bytes 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:36:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:36:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:36:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:36:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:36:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-05T22:36:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:36:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-05T22:36:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_27': 2023-12-05T22:36:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:36:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 138ac861c5eb 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.138ac861c5eb 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:50 2023 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:50 2023 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:50 2023 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d5a19c81-aa40-411c-8f9d-e68f5717dee7 2023-12-05T22:36:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:36:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:36:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-05T22:36:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:36:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:36:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:36:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:36:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:36:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:36:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:36:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 52995072 bytes 2023-12-05T22:36:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 130511872 bytes 2023-12-05T22:36:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 158969856 bytes 2023-12-05T22:36:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 184418304 bytes 2023-12-05T22:36:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 236318720 bytes 2023-12-05T22:36:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 281239552 bytes 2023-12-05T22:36:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 360170496 bytes 2023-12-05T22:36:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 429987840 bytes 2023-12-05T22:36:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 479122432 bytes 2023-12-05T22:36:53.349 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9880520357086061) > weight (0.25) 2023-12-05T22:36:53.349 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.553592988048673) > weight (0.25) 2023-12-05T22:36:53.349 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.33978888321807454) > weight (0.25) 2023-12-05T22:36:53.349 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4954160369756543) > weight (0.25) 2023-12-05T22:36:53.349 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-05T22:36:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 560077824 bytes 2023-12-05T22:36:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 611264512 bytes 2023-12-05T22:36:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 692295680 bytes 2023-12-05T22:36:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 793362432 bytes 2023-12-05T22:36:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 837896192 bytes 2023-12-05T22:36:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 839662592 bytes 2023-12-05T22:36:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 857221120 bytes 2023-12-05T22:36:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 910587904 bytes 2023-12-05T22:36:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 947636224 bytes 2023-12-05T22:36:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 967029760 bytes 2023-12-05T22:36:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1010915328 bytes 2023-12-05T22:36:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1071352832 bytes 2023-12-05T22:36:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1110323200 bytes 2023-12-05T22:36:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1173870592 bytes 2023-12-05T22:36:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1239224320 bytes 2023-12-05T22:36:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1265921024 bytes 2023-12-05T22:36:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1297890304 bytes 2023-12-05T22:36:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1342685184 bytes 2023-12-05T22:36:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1382272000 bytes 2023-12-05T22:36:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1418460160 bytes 2023-12-05T22:36:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1442944000 bytes 2023-12-05T22:36:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1531175936 bytes 2023-12-05T22:36:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1547335680 bytes 2023-12-05T22:36:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1556257792 bytes 2023-12-05T22:36:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1581499392 bytes 2023-12-05T22:36:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1593157632 bytes 2023-12-05T22:36:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1640052736 bytes 2023-12-05T22:36:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1660955648 bytes 2023-12-05T22:36:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1700301824 bytes 2023-12-05T22:36:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1700928512 bytes 2023-12-05T22:36:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1759848448 bytes 2023-12-05T22:36:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1794719744 bytes 2023-12-05T22:36:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1805727744 bytes 2023-12-05T22:36:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1832804352 bytes 2023-12-05T22:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1878755328 bytes 2023-12-05T22:36:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1895521280 bytes 2023-12-05T22:36:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1896135680 bytes 2023-12-05T22:37:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1953604608 bytes 2023-12-05T22:37:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1999551488 bytes 2023-12-05T22:37:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2056690688 bytes 2023-12-05T22:37:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2093270016 bytes 2023-12-05T22:37:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2115610624 bytes 2023-12-05T22:37:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2132625408 bytes 2023-12-05T22:37:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2182386688 bytes 2023-12-05T22:37:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2237432832 bytes 2023-12-05T22:37:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2261885952 bytes 2023-12-05T22:37:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2296754176 bytes 2023-12-05T22:37:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2325271552 bytes 2023-12-05T22:37:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2355278848 bytes 2023-12-05T22:37:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2396291072 bytes 2023-12-05T22:37:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2417146880 bytes 2023-12-05T22:37:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2431244288 bytes 2023-12-05T22:37:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2431848448 bytes 2023-12-05T22:37:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2471166976 bytes 2023-12-05T22:37:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2498571264 bytes 2023-12-05T22:37:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2559337472 bytes 2023-12-05T22:37:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2559951872 bytes 2023-12-05T22:37:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2594549760 bytes 2023-12-05T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2656926720 bytes 2023-12-05T22:37:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2681831424 bytes 2023-12-05T22:37:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2697033728 bytes 2023-12-05T22:37:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2713804800 bytes 2023-12-05T22:37:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:37:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2023-12-05T22:37:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-05T22:37:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:37:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-05T22:37:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_28': 2023-12-05T22:37:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:37:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13942b70d080 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13942b70d080 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:04 2023 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:04 2023 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:04 2023 2023-12-05T22:37:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:37:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:37:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 188e2fd3-baca-493b-8be5-724525683def 2023-12-05T22:37:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:37:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-05T22:37:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:37:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:37:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2023-12-05T22:37:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:37:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:37:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 78993408 bytes 2023-12-05T22:37:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 124514304 bytes 2023-12-05T22:37:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 134633472 bytes 2023-12-05T22:37:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 216756224 bytes 2023-12-05T22:37:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 248658944 bytes 2023-12-05T22:37:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 271483904 bytes 2023-12-05T22:37:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 363778048 bytes 2023-12-05T22:37:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 390660096 bytes 2023-12-05T22:37:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 455111680 bytes 2023-12-05T22:37:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 490422272 bytes 2023-12-05T22:37:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 541044736 bytes 2023-12-05T22:37:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 583888896 bytes 2023-12-05T22:37:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 594817024 bytes 2023-12-05T22:37:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 604508160 bytes 2023-12-05T22:37:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 646352896 bytes 2023-12-05T22:37:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 711706624 bytes 2023-12-05T22:37:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 760218624 bytes 2023-12-05T22:37:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 789888000 bytes 2023-12-05T22:37:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 793365504 bytes 2023-12-05T22:37:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 839574528 bytes 2023-12-05T22:37:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 891323392 bytes 2023-12-05T22:37:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 928269312 bytes 2023-12-05T22:37:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 961540096 bytes 2023-12-05T22:37:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 974850048 bytes 2023-12-05T22:37:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1004764160 bytes 2023-12-05T22:37:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1027588096 bytes 2023-12-05T22:37:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1077872640 bytes 2023-12-05T22:37:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1122958336 bytes 2023-12-05T22:37:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1147101184 bytes 2023-12-05T22:37:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1158162432 bytes 2023-12-05T22:37:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1218721792 bytes 2023-12-05T22:37:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1260954624 bytes 2023-12-05T22:37:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1328613376 bytes 2023-12-05T22:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1383404544 bytes 2023-12-05T22:37:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1409588224 bytes 2023-12-05T22:37:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1456274432 bytes 2023-12-05T22:37:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1508736000 bytes 2023-12-05T22:37:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1559813120 bytes 2023-12-05T22:37:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1593929728 bytes 2023-12-05T22:37:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1675670528 bytes 2023-12-05T22:37:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1689155584 bytes 2023-12-05T22:37:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2023-12-05T22:37:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:37:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-05T22:37:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:37:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_29': 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 139ee6f21e8a 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.139ee6f21e8a 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:37:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:13 2023 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:13 2023 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:13 2023 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4f42a30e-47bf-4a43-ad22-e89c1569217d 2023-12-05T22:37:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:37:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:37:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-05T22:37:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:37:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:37:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2023-12-05T22:37:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:37:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:37:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 58293248 bytes 2023-12-05T22:37:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 129009664 bytes 2023-12-05T22:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 177016832 bytes 2023-12-05T22:37:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 184824832 bytes 2023-12-05T22:37:15.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 237124608 bytes 2023-12-05T22:37:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 257295360 bytes 2023-12-05T22:37:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 297454592 bytes 2023-12-05T22:37:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 393623552 bytes 2023-12-05T22:37:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 458499072 bytes 2023-12-05T22:37:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 504426496 bytes 2023-12-05T22:37:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 559441920 bytes 2023-12-05T22:37:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 586858496 bytes 2023-12-05T22:37:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 628630528 bytes 2023-12-05T22:37:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 669443072 bytes 2023-12-05T22:37:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 719392768 bytes 2023-12-05T22:37:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 789381120 bytes 2023-12-05T22:37:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 852611072 bytes 2023-12-05T22:37:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 950364160 bytes 2023-12-05T22:37:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 950971392 bytes 2023-12-05T22:37:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 983183360 bytes 2023-12-05T22:37:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1034061824 bytes 2023-12-05T22:37:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1084540928 bytes 2023-12-05T22:37:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1096736768 bytes 2023-12-05T22:37:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1119326208 bytes 2023-12-05T22:37:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1119926272 bytes 2023-12-05T22:37:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1128889344 bytes 2023-12-05T22:37:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1157640192 bytes 2023-12-05T22:37:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1192316928 bytes 2023-12-05T22:37:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1262948352 bytes 2023-12-05T22:37:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1315299328 bytes 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:37:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:37:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-05T22:37:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:37:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-05T22:37:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_30': 2023-12-05T22:37:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13a82661e8fd 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13a82661e8fd 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:20 2023 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:20 2023 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:20 2023 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:37:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:37:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 607abc1b-0895-4b2d-abb8-f771bc5c407f 2023-12-05T22:37:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:37:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-05T22:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:37:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2023-12-05T22:37:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:37:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:37:21.350 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.36437525229048084) > weight (0.25) 2023-12-05T22:37:21.351 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6311462007144666) > weight (0.25) 2023-12-05T22:37:21.351 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-05T22:37:21.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-05T22:37:21.351 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-05T22:37:21.351 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-05T22:37:21.352 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:37:21.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:37:21.359+0000 7ff39ab1b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46016) UID: 1000 2023-12-05T22:37:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 56388608 bytes 2023-12-05T22:37:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 106413056 bytes 2023-12-05T22:37:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 139798528 bytes 2023-12-05T22:37:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 140390400 bytes 2023-12-05T22:37:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 158016512 bytes 2023-12-05T22:37:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 181906432 bytes 2023-12-05T22:37:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 283405312 bytes 2023-12-05T22:37:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 339073024 bytes 2023-12-05T22:37:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 414657536 bytes 2023-12-05T22:37:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 478430208 bytes 2023-12-05T22:37:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 481900544 bytes 2023-12-05T22:37:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 515649536 bytes 2023-12-05T22:37:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 575236096 bytes 2023-12-05T22:37:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 587744256 bytes 2023-12-05T22:37:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 646475776 bytes 2023-12-05T22:37:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 745831424 bytes 2023-12-05T22:37:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 776887296 bytes 2023-12-05T22:37:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 842114048 bytes 2023-12-05T22:37:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 890972160 bytes 2023-12-05T22:37:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 901201920 bytes 2023-12-05T22:37:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 901797888 bytes 2023-12-05T22:37:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 960247808 bytes 2023-12-05T22:37:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 973097984 bytes 2023-12-05T22:37:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 973697024 bytes 2023-12-05T22:37:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1014185984 bytes 2023-12-05T22:37:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1068723200 bytes 2023-12-05T22:37:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1101998080 bytes 2023-12-05T22:37:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1172848640 bytes 2023-12-05T22:37:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1241054208 bytes 2023-12-05T22:37:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1332957184 bytes 2023-12-05T22:37:27.455 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-05T22:37:27.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-05T22:37:27.455 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-05T22:37:27.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-05T22:37:27.458 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-05T22:37:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1384602624 bytes 2023-12-05T22:37:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1390483456 bytes 2023-12-05T22:37:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1445123072 bytes 2023-12-05T22:37:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1527540736 bytes 2023-12-05T22:37:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:37:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:37:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-05T22:37:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:37:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-05T22:37:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_31': 2023-12-05T22:37:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:37:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:37:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:37:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13bab8a8003c 2023-12-05T22:37:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13bab8a8003c 2023-12-05T22:37:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:28 2023 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:28 2023 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:28 2023 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 36786ed1-b139-4c21-bcaa-0cacd1cad4dc 2023-12-05T22:37:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:37:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:37:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-05T22:37:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:37:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:37:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:37:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:37:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:37:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 72352768 bytes 2023-12-05T22:37:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 118142976 bytes 2023-12-05T22:37:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 177315840 bytes 2023-12-05T22:37:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 228860928 bytes 2023-12-05T22:37:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 251558912 bytes 2023-12-05T22:37:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 269725696 bytes 2023-12-05T22:37:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 329230336 bytes 2023-12-05T22:37:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 424168448 bytes 2023-12-05T22:37:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 474169344 bytes 2023-12-05T22:37:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 535062528 bytes 2023-12-05T22:37:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 619417600 bytes 2023-12-05T22:37:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 676116480 bytes 2023-12-05T22:37:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 703071232 bytes 2023-12-05T22:37:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 817256448 bytes 2023-12-05T22:37:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 886295552 bytes 2023-12-05T22:37:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 950560768 bytes 2023-12-05T22:37:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 975947776 bytes 2023-12-05T22:37:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 1043859456 bytes 2023-12-05T22:37:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1073115136 bytes 2023-12-05T22:37:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1073730560 bytes 2023-12-05T22:37:33.710 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:37:33.715+0000 7fe8efb13640 -1 reset not still connected to 0x55b60f20b380 2023-12-05T22:37:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1097577472 bytes 2023-12-05T22:37:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1140572160 bytes 2023-12-05T22:37:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1191088128 bytes 2023-12-05T22:37:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1191689216 bytes 2023-12-05T22:37:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1262236672 bytes 2023-12-05T22:37:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1347652608 bytes 2023-12-05T22:37:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1393910784 bytes 2023-12-05T22:37:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1491453952 bytes 2023-12-05T22:37:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1572155392 bytes 2023-12-05T22:37:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1592484864 bytes 2023-12-05T22:37:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1636027392 bytes 2023-12-05T22:37:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1675415552 bytes 2023-12-05T22:37:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1677536256 bytes 2023-12-05T22:37:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1739506688 bytes 2023-12-05T22:37:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1841183744 bytes 2023-12-05T22:37:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1866968064 bytes 2023-12-05T22:37:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1948485632 bytes 2023-12-05T22:37:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1991657472 bytes 2023-12-05T22:37:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1994486784 bytes 2023-12-05T22:37:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2013980672 bytes 2023-12-05T22:37:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2069601280 bytes 2023-12-05T22:37:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2096313344 bytes 2023-12-05T22:37:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2168473600 bytes 2023-12-05T22:37:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2208100352 bytes 2023-12-05T22:37:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2229287936 bytes 2023-12-05T22:37:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2279091200 bytes 2023-12-05T22:37:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2300787712 bytes 2023-12-05T22:37:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2314486784 bytes 2023-12-05T22:37:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2365763584 bytes 2023-12-05T22:37:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2458234880 bytes 2023-12-05T22:37:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2535058432 bytes 2023-12-05T22:37:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2570377216 bytes 2023-12-05T22:37:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2605826048 bytes 2023-12-05T22:37:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2616991744 bytes 2023-12-05T22:37:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2694918144 bytes 2023-12-05T22:37:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2719181824 bytes 2023-12-05T22:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2769420288 bytes 2023-12-05T22:37:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2831260672 bytes 2023-12-05T22:37:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2879599616 bytes 2023-12-05T22:37:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2926854144 bytes 2023-12-05T22:37:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2969681920 bytes 2023-12-05T22:37:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2999476224 bytes 2023-12-05T22:37:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 3053797376 bytes 2023-12-05T22:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 3073384448 bytes 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:37:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-05T22:37:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:37:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_32': 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13c36850c9b3 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13c36850c9b3 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:42 2023 2023-12-05T22:37:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:42 2023 2023-12-05T22:37:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:42 2023 2023-12-05T22:37:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:37:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:37:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 726117d0-c60f-4f67-b079-12119b0741a6 2023-12-05T22:37:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:37:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:37:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-05T22:37:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:37:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:37:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2023-12-05T22:37:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:37:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:37:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 19936256 bytes 2023-12-05T22:37:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 26218496 bytes 2023-12-05T22:37:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 65370112 bytes 2023-12-05T22:37:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 98401280 bytes 2023-12-05T22:37:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 132982784 bytes 2023-12-05T22:37:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 152668160 bytes 2023-12-05T22:37:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 213158912 bytes 2023-12-05T22:37:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 250053632 bytes 2023-12-05T22:37:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 250675200 bytes 2023-12-05T22:37:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 283806720 bytes 2023-12-05T22:37:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 308292608 bytes 2023-12-05T22:37:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 390380544 bytes 2023-12-05T22:37:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 397217792 bytes 2023-12-05T22:37:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 460527616 bytes 2023-12-05T22:37:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 516557824 bytes 2023-12-05T22:37:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 535798784 bytes 2023-12-05T22:37:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 606764032 bytes 2023-12-05T22:37:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 642620416 bytes 2023-12-05T22:37:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 683480064 bytes 2023-12-05T22:37:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 743990272 bytes 2023-12-05T22:37:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 796083200 bytes 2023-12-05T22:37:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 851949568 bytes 2023-12-05T22:37:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 951363584 bytes 2023-12-05T22:37:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 988794880 bytes 2023-12-05T22:37:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1011373056 bytes 2023-12-05T22:37:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1055116288 bytes 2023-12-05T22:37:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1071672320 bytes 2023-12-05T22:37:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:37:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2023-12-05T22:37:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:37:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:37:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:37:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-05T22:37:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:37:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:37:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-05T22:37:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:37:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_33': 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13cdcdd99908 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13cdcdd99908 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:37:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:48 2023 2023-12-05T22:37:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:48 2023 2023-12-05T22:37:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:48 2023 2023-12-05T22:37:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:37:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:37:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 80cd4891-b460-4331-9eae-304b773fde31 2023-12-05T22:37:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:37:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:37:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-05T22:37:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:37:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:37:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:37:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:37:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:37:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:37:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:37:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 37919744 bytes 2023-12-05T22:37:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 129341440 bytes 2023-12-05T22:37:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 155258880 bytes 2023-12-05T22:37:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 155865088 bytes 2023-12-05T22:37:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 220658688 bytes 2023-12-05T22:37:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 254910464 bytes 2023-12-05T22:37:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 255522816 bytes 2023-12-05T22:37:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 256149504 bytes 2023-12-05T22:37:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 296215552 bytes 2023-12-05T22:37:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 315247616 bytes 2023-12-05T22:37:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 320144384 bytes 2023-12-05T22:37:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 372754432 bytes 2023-12-05T22:37:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 444267520 bytes 2023-12-05T22:37:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 483606528 bytes 2023-12-05T22:37:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 553274368 bytes 2023-12-05T22:37:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 553895936 bytes 2023-12-05T22:37:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 606664704 bytes 2023-12-05T22:37:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 664229888 bytes 2023-12-05T22:37:51.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 705993728 bytes 2023-12-05T22:37:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 760684544 bytes 2023-12-05T22:37:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 785880064 bytes 2023-12-05T22:37:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 841060352 bytes 2023-12-05T22:37:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 860450816 bytes 2023-12-05T22:37:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 903320576 bytes 2023-12-05T22:37:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 918374400 bytes 2023-12-05T22:37:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 971582464 bytes 2023-12-05T22:37:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1068013568 bytes 2023-12-05T22:37:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1117147136 bytes 2023-12-05T22:37:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1191519232 bytes 2023-12-05T22:37:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1269811200 bytes 2023-12-05T22:37:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1317332992 bytes 2023-12-05T22:37:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1359219712 bytes 2023-12-05T22:37:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1422074880 bytes 2023-12-05T22:37:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1443382272 bytes 2023-12-05T22:37:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1508497408 bytes 2023-12-05T22:37:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1547809792 bytes 2023-12-05T22:37:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1629075456 bytes 2023-12-05T22:37:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1681192960 bytes 2023-12-05T22:37:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1728200704 bytes 2023-12-05T22:37:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1745991680 bytes 2023-12-05T22:37:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1765301248 bytes 2023-12-05T22:37:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1794037760 bytes 2023-12-05T22:37:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1868103680 bytes 2023-12-05T22:37:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1888784384 bytes 2023-12-05T22:37:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1952551936 bytes 2023-12-05T22:37:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1976511488 bytes 2023-12-05T22:37:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2017510400 bytes 2023-12-05T22:37:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2042844160 bytes 2023-12-05T22:37:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2088850432 bytes 2023-12-05T22:37:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2132375552 bytes 2023-12-05T22:37:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2154677248 bytes 2023-12-05T22:37:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2208317440 bytes 2023-12-05T22:37:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2232479744 bytes 2023-12-05T22:37:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2275777536 bytes 2023-12-05T22:37:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2300055552 bytes 2023-12-05T22:37:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2348054528 bytes 2023-12-05T22:37:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2374084608 bytes 2023-12-05T22:37:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2395111424 bytes 2023-12-05T22:37:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2420799488 bytes 2023-12-05T22:37:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2468465664 bytes 2023-12-05T22:38:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2492209152 bytes 2023-12-05T22:38:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2504519680 bytes 2023-12-05T22:38:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2521733120 bytes 2023-12-05T22:38:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2556509184 bytes 2023-12-05T22:38:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2693462016 bytes 2023-12-05T22:38:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2698254336 bytes 2023-12-05T22:38:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2750332928 bytes 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:38:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2023-12-05T22:38:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-05T22:38:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:38:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_34': 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13d678bb81c1 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13d678bb81c1 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:38:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:38:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:01 2023 2023-12-05T22:38:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:01 2023 2023-12-05T22:38:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:01 2023 2023-12-05T22:38:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:38:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:38:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c95cf52a-6e99-4b13-94b9-00efacfbc9bf 2023-12-05T22:38:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:38:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:38:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-05T22:38:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:38:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:38:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:38:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:38:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:38:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 54176768 bytes 2023-12-05T22:38:02.350 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.34134833626719396) > weight (0.25) 2023-12-05T22:38:02.351 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.48000509580110795) > weight (0.25) 2023-12-05T22:38:02.351 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-05T22:38:02.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-05T22:38:02.351 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-05T22:38:02.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:38:02.358+0000 7f321f8d6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 49334) UID: 1000 2023-12-05T22:38:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 82154496 bytes 2023-12-05T22:38:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 172017664 bytes 2023-12-05T22:38:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 229718016 bytes 2023-12-05T22:38:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 265680896 bytes 2023-12-05T22:38:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 310027264 bytes 2023-12-05T22:38:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 419805184 bytes 2023-12-05T22:38:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 463098880 bytes 2023-12-05T22:38:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 529886208 bytes 2023-12-05T22:38:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 546531328 bytes 2023-12-05T22:38:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 569240576 bytes 2023-12-05T22:38:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 602825728 bytes 2023-12-05T22:38:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 625602560 bytes 2023-12-05T22:38:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 653447168 bytes 2023-12-05T22:38:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 695072768 bytes 2023-12-05T22:38:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 748919808 bytes 2023-12-05T22:38:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 821982208 bytes 2023-12-05T22:38:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 851701760 bytes 2023-12-05T22:38:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 870424576 bytes 2023-12-05T22:38:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 932902912 bytes 2023-12-05T22:38:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 945598464 bytes 2023-12-05T22:38:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1033628672 bytes 2023-12-05T22:38:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1055965184 bytes 2023-12-05T22:38:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1073542144 bytes 2023-12-05T22:38:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1119205376 bytes 2023-12-05T22:38:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1190603776 bytes 2023-12-05T22:38:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1192907776 bytes 2023-12-05T22:38:07.352 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-05T22:38:07.353 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:38:07.353 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-05T22:38:07.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-05T22:38:07.353 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-05T22:38:07.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-05T22:38:07.356 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-05T22:38:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1239363584 bytes 2023-12-05T22:38:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1289298944 bytes 2023-12-05T22:38:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1337028608 bytes 2023-12-05T22:38:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1337621504 bytes 2023-12-05T22:38:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1388690432 bytes 2023-12-05T22:38:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1511347200 bytes 2023-12-05T22:38:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1586473984 bytes 2023-12-05T22:38:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1587072000 bytes 2023-12-05T22:38:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1652269056 bytes 2023-12-05T22:38:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1716940800 bytes 2023-12-05T22:38:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1750771712 bytes 2023-12-05T22:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1768548352 bytes 2023-12-05T22:38:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1796548608 bytes 2023-12-05T22:38:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1829695488 bytes 2023-12-05T22:38:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1873284096 bytes 2023-12-05T22:38:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1881968640 bytes 2023-12-05T22:38:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1915385856 bytes 2023-12-05T22:38:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1915980800 bytes 2023-12-05T22:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1933243392 bytes 2023-12-05T22:38:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1992411136 bytes 2023-12-05T22:38:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2042479616 bytes 2023-12-05T22:38:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2090420224 bytes 2023-12-05T22:38:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2111302656 bytes 2023-12-05T22:38:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2138872832 bytes 2023-12-05T22:38:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2161400832 bytes 2023-12-05T22:38:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2227246080 bytes 2023-12-05T22:38:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2264336384 bytes 2023-12-05T22:38:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2317346816 bytes 2023-12-05T22:38:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2376181760 bytes 2023-12-05T22:38:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2376801280 bytes 2023-12-05T22:38:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2442164224 bytes 2023-12-05T22:38:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2543194112 bytes 2023-12-05T22:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2613082112 bytes 2023-12-05T22:38:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2640233472 bytes 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:38:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:38:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2023-12-05T22:38:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:38:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:38:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-05T22:38:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:38:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-05T22:38:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:38:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_35': 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13e4f54921a1 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13e4f54921a1 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:38:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:14 2023 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:14 2023 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:14 2023 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1d6600a5-89f1-4e94-812c-f425e83f649a 2023-12-05T22:38:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:38:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:38:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-05T22:38:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:38:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:38:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:38:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:38:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:38:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 91297792 bytes 2023-12-05T22:38:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 154020864 bytes 2023-12-05T22:38:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 194247680 bytes 2023-12-05T22:38:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 250551296 bytes 2023-12-05T22:38:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 269388800 bytes 2023-12-05T22:38:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 293874688 bytes 2023-12-05T22:38:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 311444480 bytes 2023-12-05T22:38:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 319662080 bytes 2023-12-05T22:38:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 381111296 bytes 2023-12-05T22:38:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 397310976 bytes 2023-12-05T22:38:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 504205312 bytes 2023-12-05T22:38:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 553139200 bytes 2023-12-05T22:38:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 625344512 bytes 2023-12-05T22:38:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 697539584 bytes 2023-12-05T22:38:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 715672576 bytes 2023-12-05T22:38:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 786406400 bytes 2023-12-05T22:38:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 804483072 bytes 2023-12-05T22:38:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 833573888 bytes 2023-12-05T22:38:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 898285568 bytes 2023-12-05T22:38:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 898894848 bytes 2023-12-05T22:38:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 968384512 bytes 2023-12-05T22:38:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1075685376 bytes 2023-12-05T22:38:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1166258176 bytes 2023-12-05T22:38:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1208250368 bytes 2023-12-05T22:38:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1283677184 bytes 2023-12-05T22:38:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1290750976 bytes 2023-12-05T22:38:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1297503232 bytes 2023-12-05T22:38:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1372409856 bytes 2023-12-05T22:38:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1462064128 bytes 2023-12-05T22:38:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1481196544 bytes 2023-12-05T22:38:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1491441664 bytes 2023-12-05T22:38:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1536472064 bytes 2023-12-05T22:38:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1592876032 bytes 2023-12-05T22:38:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1625957376 bytes 2023-12-05T22:38:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1670002688 bytes 2023-12-05T22:38:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1717245952 bytes 2023-12-05T22:38:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1742510080 bytes 2023-12-05T22:38:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1826433024 bytes 2023-12-05T22:38:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1895906304 bytes 2023-12-05T22:38:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1901546496 bytes 2023-12-05T22:38:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1931838464 bytes 2023-12-05T22:38:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1973316608 bytes 2023-12-05T22:38:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2017465344 bytes 2023-12-05T22:38:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2076779520 bytes 2023-12-05T22:38:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2107176960 bytes 2023-12-05T22:38:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2195616768 bytes 2023-12-05T22:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2227442688 bytes 2023-12-05T22:38:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2249757696 bytes 2023-12-05T22:38:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2276936704 bytes 2023-12-05T22:38:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2315708416 bytes 2023-12-05T22:38:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2361289728 bytes 2023-12-05T22:38:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2438523904 bytes 2023-12-05T22:38:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2445221888 bytes 2023-12-05T22:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2490285056 bytes 2023-12-05T22:38:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2500392960 bytes 2023-12-05T22:38:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2539732992 bytes 2023-12-05T22:38:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2559062016 bytes 2023-12-05T22:38:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2560366592 bytes 2023-12-05T22:38:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2602174464 bytes 2023-12-05T22:38:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2612230144 bytes 2023-12-05T22:38:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2655003648 bytes 2023-12-05T22:38:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2709672960 bytes 2023-12-05T22:38:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2746663936 bytes 2023-12-05T22:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2800953344 bytes 2023-12-05T22:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2804989952 bytes 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:38:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:38:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:38:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2023-12-05T22:38:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:38:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:38:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-05T22:38:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:38:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-05T22:38:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:38:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_36': 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13ec13d27081 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13ec13d27081 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:38:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:27 2023 2023-12-05T22:38:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:27 2023 2023-12-05T22:38:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:27 2023 2023-12-05T22:38:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:38:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:38:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4f5aa172-1810-4fc1-84bc-bfbb3a129dcd 2023-12-05T22:38:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:38:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:38:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-05T22:38:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:38:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:38:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2023-12-05T22:38:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:38:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 12839936 bytes 2023-12-05T22:38:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 28907520 bytes 2023-12-05T22:38:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 88363008 bytes 2023-12-05T22:38:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 153151488 bytes 2023-12-05T22:38:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 198721536 bytes 2023-12-05T22:38:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 260550656 bytes 2023-12-05T22:38:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 288026624 bytes 2023-12-05T22:38:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 318570496 bytes 2023-12-05T22:38:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 353537024 bytes 2023-12-05T22:38:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 382369792 bytes 2023-12-05T22:38:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 448731136 bytes 2023-12-05T22:38:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 546616320 bytes 2023-12-05T22:38:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 575682560 bytes 2023-12-05T22:38:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 632636416 bytes 2023-12-05T22:38:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 676347904 bytes 2023-12-05T22:38:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 676942848 bytes 2023-12-05T22:38:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 712640512 bytes 2023-12-05T22:38:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 744098816 bytes 2023-12-05T22:38:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 751883264 bytes 2023-12-05T22:38:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 798575616 bytes 2023-12-05T22:38:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 826302464 bytes 2023-12-05T22:38:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 856949760 bytes 2023-12-05T22:38:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 930672640 bytes 2023-12-05T22:38:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 949089280 bytes 2023-12-05T22:38:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 975211520 bytes 2023-12-05T22:38:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 997362688 bytes 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:38:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:38:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-05T22:38:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:38:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_37': 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13f68cf5b3a7 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13f68cf5b3a7 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:38:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:33 2023 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:33 2023 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:33 2023 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 21086603-d365-40b9-9a1b-a87abe17542f 2023-12-05T22:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:38:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:38:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-05T22:38:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:38:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:38:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2023-12-05T22:38:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:38:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:38:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 55448576 bytes 2023-12-05T22:38:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 107930624 bytes 2023-12-05T22:38:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 146985984 bytes 2023-12-05T22:38:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 196782080 bytes 2023-12-05T22:38:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 273637376 bytes 2023-12-05T22:38:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 285015040 bytes 2023-12-05T22:38:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 293694464 bytes 2023-12-05T22:38:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 402696192 bytes 2023-12-05T22:38:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 423434240 bytes 2023-12-05T22:38:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 454384640 bytes 2023-12-05T22:38:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 474380288 bytes 2023-12-05T22:38:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 501095424 bytes 2023-12-05T22:38:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 529175552 bytes 2023-12-05T22:38:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 547953664 bytes 2023-12-05T22:38:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 551670784 bytes 2023-12-05T22:38:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 631085056 bytes 2023-12-05T22:38:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 674863104 bytes 2023-12-05T22:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 741339136 bytes 2023-12-05T22:38:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 818137088 bytes 2023-12-05T22:38:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 863138816 bytes 2023-12-05T22:38:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 911439872 bytes 2023-12-05T22:38:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 962124800 bytes 2023-12-05T22:38:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1032946688 bytes 2023-12-05T22:38:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1057869824 bytes 2023-12-05T22:38:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1124961280 bytes 2023-12-05T22:38:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1205606400 bytes 2023-12-05T22:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1206199296 bytes 2023-12-05T22:38:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1262626816 bytes 2023-12-05T22:38:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1308283904 bytes 2023-12-05T22:38:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1339182080 bytes 2023-12-05T22:38:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1416695808 bytes 2023-12-05T22:38:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1517469696 bytes 2023-12-05T22:38:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1518078976 bytes 2023-12-05T22:38:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1527282688 bytes 2023-12-05T22:38:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1569618944 bytes 2023-12-05T22:38:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1579563008 bytes 2023-12-05T22:38:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1661404160 bytes 2023-12-05T22:38:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1685509120 bytes 2023-12-05T22:38:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1717989376 bytes 2023-12-05T22:38:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1728933888 bytes 2023-12-05T22:38:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1729552384 bytes 2023-12-05T22:38:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1747220480 bytes 2023-12-05T22:38:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1804694528 bytes 2023-12-05T22:38:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1822184448 bytes 2023-12-05T22:38:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1886900224 bytes 2023-12-05T22:38:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1917637632 bytes 2023-12-05T22:38:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1918263296 bytes 2023-12-05T22:38:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1972422656 bytes 2023-12-05T22:38:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2058605568 bytes 2023-12-05T22:38:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2059226112 bytes 2023-12-05T22:38:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2090850304 bytes 2023-12-05T22:38:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2140021760 bytes 2023-12-05T22:38:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2180285440 bytes 2023-12-05T22:38:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2202432512 bytes 2023-12-05T22:38:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2250401792 bytes 2023-12-05T22:38:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2294623232 bytes 2023-12-05T22:38:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2328850432 bytes 2023-12-05T22:38:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-05T22:38:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:38:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-05T22:38:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_38': 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1401e20ab256 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1401e20ab256 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:45 2023 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:45 2023 2023-12-05T22:38:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:45 2023 2023-12-05T22:38:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:38:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:38:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ffdb2cf0-ec49-4f59-9404-9cf65a1b49e0 2023-12-05T22:38:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:38:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:38:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-05T22:38:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:38:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:38:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:38:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:38:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:38:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 70444032 bytes 2023-12-05T22:38:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 145321984 bytes 2023-12-05T22:38:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 212322304 bytes 2023-12-05T22:38:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 243186688 bytes 2023-12-05T22:38:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 286608384 bytes 2023-12-05T22:38:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 318921728 bytes 2023-12-05T22:38:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 379855872 bytes 2023-12-05T22:38:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 396393472 bytes 2023-12-05T22:38:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 433880064 bytes 2023-12-05T22:38:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 454371328 bytes 2023-12-05T22:38:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 498677760 bytes 2023-12-05T22:38:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 608393216 bytes 2023-12-05T22:38:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 646529024 bytes 2023-12-05T22:38:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 717574144 bytes 2023-12-05T22:38:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 746817536 bytes 2023-12-05T22:38:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 784437248 bytes 2023-12-05T22:38:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 813329408 bytes 2023-12-05T22:38:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 897340416 bytes 2023-12-05T22:38:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 909752320 bytes 2023-12-05T22:38:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 935431168 bytes 2023-12-05T22:38:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 990945280 bytes 2023-12-05T22:38:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1052906496 bytes 2023-12-05T22:38:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1131207680 bytes 2023-12-05T22:38:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1164250112 bytes 2023-12-05T22:38:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1245669376 bytes 2023-12-05T22:38:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1300632576 bytes 2023-12-05T22:38:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1352411136 bytes 2023-12-05T22:38:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1367133184 bytes 2023-12-05T22:38:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1399223296 bytes 2023-12-05T22:38:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1446194176 bytes 2023-12-05T22:38:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1456123904 bytes 2023-12-05T22:38:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1483953152 bytes 2023-12-05T22:38:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1508388864 bytes 2023-12-05T22:38:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1563320320 bytes 2023-12-05T22:38:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1616601088 bytes 2023-12-05T22:38:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1657330688 bytes 2023-12-05T22:38:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1748661248 bytes 2023-12-05T22:38:54.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1782276096 bytes 2023-12-05T22:38:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1828114432 bytes 2023-12-05T22:38:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1845783552 bytes 2023-12-05T22:38:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1922378752 bytes 2023-12-05T22:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1980695552 bytes 2023-12-05T22:38:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2039572480 bytes 2023-12-05T22:38:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2058112000 bytes 2023-12-05T22:38:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2109017088 bytes 2023-12-05T22:38:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2156058624 bytes 2023-12-05T22:38:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2229745664 bytes 2023-12-05T22:38:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2260072448 bytes 2023-12-05T22:38:56.459 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6302978477101879) > weight (0.25) 2023-12-05T22:38:56.459 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4662297238864863) > weight (0.25) 2023-12-05T22:38:56.459 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9875283497782651) > weight (0.25) 2023-12-05T22:38:56.459 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6722244115144003) > weight (0.25) 2023-12-05T22:38:56.460 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-05T22:38:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2303353856 bytes 2023-12-05T22:38:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2344408064 bytes 2023-12-05T22:38:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2399772672 bytes 2023-12-05T22:38:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2419857408 bytes 2023-12-05T22:38:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2505325568 bytes 2023-12-05T22:38:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2556869632 bytes 2023-12-05T22:38:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2625625088 bytes 2023-12-05T22:38:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2654579712 bytes 2023-12-05T22:38:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2664742912 bytes 2023-12-05T22:38:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:38:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2023-12-05T22:38:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:38:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-05T22:38:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:38:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-05T22:38:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_39': 2023-12-05T22:38:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 140b5459318b 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.140b5459318b 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:58 2023 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:58 2023 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:58 2023 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:38:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:38:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b11ea2a8-7502-44da-ba95-ff96ab7a913f 2023-12-05T22:38:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:38:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:38:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-05T22:38:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:38:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:38:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:38:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:38:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2023-12-05T22:38:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:38:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:38:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 38577152 bytes 2023-12-05T22:38:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 73808896 bytes 2023-12-05T22:38:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 110795776 bytes 2023-12-05T22:39:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 156620800 bytes 2023-12-05T22:39:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 206397440 bytes 2023-12-05T22:39:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 249973760 bytes 2023-12-05T22:39:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 356287488 bytes 2023-12-05T22:39:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 397148160 bytes 2023-12-05T22:39:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 458178560 bytes 2023-12-05T22:39:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 525468672 bytes 2023-12-05T22:39:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 555012096 bytes 2023-12-05T22:39:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 575604736 bytes 2023-12-05T22:39:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 590121984 bytes 2023-12-05T22:39:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 590747648 bytes 2023-12-05T22:39:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 604548096 bytes 2023-12-05T22:39:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 630687744 bytes 2023-12-05T22:39:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 696646656 bytes 2023-12-05T22:39:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 744365056 bytes 2023-12-05T22:39:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 751479808 bytes 2023-12-05T22:39:03.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 770171904 bytes 2023-12-05T22:39:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 835493888 bytes 2023-12-05T22:39:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 901442560 bytes 2023-12-05T22:39:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1001892864 bytes 2023-12-05T22:39:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1026617344 bytes 2023-12-05T22:39:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1062564864 bytes 2023-12-05T22:39:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1075283968 bytes 2023-12-05T22:39:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1111964672 bytes 2023-12-05T22:39:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1118219264 bytes 2023-12-05T22:39:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1162884096 bytes 2023-12-05T22:39:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1201582080 bytes 2023-12-05T22:39:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1264974848 bytes 2023-12-05T22:39:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:39:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:39:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:39:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-05T22:39:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:39:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-05T22:39:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_40': 2023-12-05T22:39:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:39:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 141435b46485 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.141435b46485 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:05 2023 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:05 2023 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:05 2023 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 8480990e-2ffd-4b7c-89de-bb66a57203df 2023-12-05T22:39:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:39:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:39:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-05T22:39:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:39:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:39:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:39:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:39:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:39:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 49299456 bytes 2023-12-05T22:39:06.719 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:06.728+0000 7fdd570d4640 -1 reset not still connected to 0x559d285fdc70 2023-12-05T22:39:06.721 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:06.728+0000 7fdd570d4640 -1 reset not still connected to 0x559d29e60dd0 2023-12-05T22:39:06.721 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:06.728+0000 7fdd570d4640 -1 reset not still connected to 0x559d2dc33040 2023-12-05T22:39:06.721 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:06.728+0000 7fdd570d4640 -1 reset not still connected to 0x559d59913380 2023-12-05T22:39:06.721 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:06.728+0000 7fdd570d4640 -1 reset not still connected to 0x559d671d1ba0 2023-12-05T22:39:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 108778496 bytes 2023-12-05T22:39:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 132777984 bytes 2023-12-05T22:39:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 152417280 bytes 2023-12-05T22:39:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 207546368 bytes 2023-12-05T22:39:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 255399936 bytes 2023-12-05T22:39:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 303106048 bytes 2023-12-05T22:39:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 307789824 bytes 2023-12-05T22:39:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 405592064 bytes 2023-12-05T22:39:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 468051968 bytes 2023-12-05T22:39:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 523503616 bytes 2023-12-05T22:39:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 575772672 bytes 2023-12-05T22:39:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 643907584 bytes 2023-12-05T22:39:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 703996928 bytes 2023-12-05T22:39:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 739950592 bytes 2023-12-05T22:39:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 873371648 bytes 2023-12-05T22:39:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 951611392 bytes 2023-12-05T22:39:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 1023745024 bytes 2023-12-05T22:39:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1057637376 bytes 2023-12-05T22:39:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1101005824 bytes 2023-12-05T22:39:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1138754560 bytes 2023-12-05T22:39:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1139361792 bytes 2023-12-05T22:39:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1225982976 bytes 2023-12-05T22:39:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1290625024 bytes 2023-12-05T22:39:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1335424000 bytes 2023-12-05T22:39:11.814 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:11.819+0000 7fdd570d4640 -1 reset not still connected to 0x559d2a678000 2023-12-05T22:39:11.814 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:11.819+0000 7fdd570d4640 -1 reset not still connected to 0x559d2e415110 2023-12-05T22:39:11.814 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:11.819+0000 7fdd570d4640 -1 reset not still connected to 0x559d2f14fba0 2023-12-05T22:39:11.814 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:39:11.819+0000 7fdd570d4640 -1 reset not still connected to 0x559d699cd6c0 2023-12-05T22:39:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1381058560 bytes 2023-12-05T22:39:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1406314496 bytes 2023-12-05T22:39:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1467300864 bytes 2023-12-05T22:39:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1550631936 bytes 2023-12-05T22:39:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1625597952 bytes 2023-12-05T22:39:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1651376128 bytes 2023-12-05T22:39:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1718816768 bytes 2023-12-05T22:39:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1823949824 bytes 2023-12-05T22:39:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1870192640 bytes 2023-12-05T22:39:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1918098432 bytes 2023-12-05T22:39:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1979131904 bytes 2023-12-05T22:39:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 2032412672 bytes 2023-12-05T22:39:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 2049951744 bytes 2023-12-05T22:39:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 2076024832 bytes 2023-12-05T22:39:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2141772800 bytes 2023-12-05T22:39:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2156316672 bytes 2023-12-05T22:39:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2242399232 bytes 2023-12-05T22:39:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2289614848 bytes 2023-12-05T22:39:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2360643584 bytes 2023-12-05T22:39:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2374156288 bytes 2023-12-05T22:39:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2422201344 bytes 2023-12-05T22:39:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2462531584 bytes 2023-12-05T22:39:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2483057664 bytes 2023-12-05T22:39:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2521967616 bytes 2023-12-05T22:39:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2585817088 bytes 2023-12-05T22:39:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2608991232 bytes 2023-12-05T22:39:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2614303744 bytes 2023-12-05T22:39:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2643706880 bytes 2023-12-05T22:39:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2698756096 bytes 2023-12-05T22:39:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2791149568 bytes 2023-12-05T22:39:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2826565632 bytes 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:39:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:39:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-05T22:39:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:39:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-05T22:39:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_41': 2023-12-05T22:39:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:39:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:39:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:39:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 141c7d230ea4 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.141c7d230ea4 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:18 2023 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:18 2023 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:18 2023 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f48ece28-d198-4bf2-878b-dd40720a4c29 2023-12-05T22:39:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:39:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:39:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-05T22:39:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:39:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:39:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:39:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:39:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:39:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 37341184 bytes 2023-12-05T22:39:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 83955712 bytes 2023-12-05T22:39:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 130329600 bytes 2023-12-05T22:39:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 133654528 bytes 2023-12-05T22:39:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 200765440 bytes 2023-12-05T22:39:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 233892864 bytes 2023-12-05T22:39:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 293390336 bytes 2023-12-05T22:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 322342912 bytes 2023-12-05T22:39:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 398208000 bytes 2023-12-05T22:39:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 422429696 bytes 2023-12-05T22:39:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 446026752 bytes 2023-12-05T22:39:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 568471552 bytes 2023-12-05T22:39:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 626310144 bytes 2023-12-05T22:39:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 662363136 bytes 2023-12-05T22:39:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 662981632 bytes 2023-12-05T22:39:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 732940288 bytes 2023-12-05T22:39:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 758437888 bytes 2023-12-05T22:39:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 808500224 bytes 2023-12-05T22:39:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 828923904 bytes 2023-12-05T22:39:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 906932224 bytes 2023-12-05T22:39:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 907531264 bytes 2023-12-05T22:39:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 978646016 bytes 2023-12-05T22:39:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1097659392 bytes 2023-12-05T22:39:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1126005760 bytes 2023-12-05T22:39:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1172242432 bytes 2023-12-05T22:39:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1172873216 bytes 2023-12-05T22:39:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1178052608 bytes 2023-12-05T22:39:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1195107328 bytes 2023-12-05T22:39:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1226274816 bytes 2023-12-05T22:39:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1255017472 bytes 2023-12-05T22:39:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1334538240 bytes 2023-12-05T22:39:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1396432896 bytes 2023-12-05T22:39:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1415187456 bytes 2023-12-05T22:39:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1459358720 bytes 2023-12-05T22:39:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1528752128 bytes 2023-12-05T22:39:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1538846720 bytes 2023-12-05T22:39:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1590364160 bytes 2023-12-05T22:39:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1609626624 bytes 2023-12-05T22:39:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1677231104 bytes 2023-12-05T22:39:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1747935232 bytes 2023-12-05T22:39:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1782916096 bytes 2023-12-05T22:39:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1788053504 bytes 2023-12-05T22:39:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1801446400 bytes 2023-12-05T22:39:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1849873408 bytes 2023-12-05T22:39:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1850472448 bytes 2023-12-05T22:39:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1897865216 bytes 2023-12-05T22:39:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1952596992 bytes 2023-12-05T22:39:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1974503424 bytes 2023-12-05T22:39:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2016427008 bytes 2023-12-05T22:39:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2039063552 bytes 2023-12-05T22:39:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2067859456 bytes 2023-12-05T22:39:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2098363392 bytes 2023-12-05T22:39:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2116114432 bytes 2023-12-05T22:39:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2169281536 bytes 2023-12-05T22:39:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2192905216 bytes 2023-12-05T22:39:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2203982848 bytes 2023-12-05T22:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2244428800 bytes 2023-12-05T22:39:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2259569664 bytes 2023-12-05T22:39:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2332984320 bytes 2023-12-05T22:39:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2353758208 bytes 2023-12-05T22:39:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2369089536 bytes 2023-12-05T22:39:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2427440128 bytes 2023-12-05T22:39:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2468152320 bytes 2023-12-05T22:39:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2522657792 bytes 2023-12-05T22:39:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2561713152 bytes 2023-12-05T22:39:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2691110912 bytes 2023-12-05T22:39:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2751177728 bytes 2023-12-05T22:39:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2797059072 bytes 2023-12-05T22:39:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2871188480 bytes 2023-12-05T22:39:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2909857792 bytes 2023-12-05T22:39:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2929454080 bytes 2023-12-05T22:39:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2930051072 bytes 2023-12-05T22:39:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2935505920 bytes 2023-12-05T22:39:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2940818432 bytes 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:39:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2023-12-05T22:39:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:39:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-05T22:39:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:39:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-05T22:39:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_42': 2023-12-05T22:39:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:39:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:39:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 142444465978 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.142444465978 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:33 2023 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:33 2023 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:33 2023 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:39:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 50eb144e-0936-49e6-8545-fb513c5ca376 2023-12-05T22:39:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:39:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:39:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-05T22:39:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:39:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:39:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2023-12-05T22:39:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:39:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:39:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 9448448 bytes 2023-12-05T22:39:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 15187968 bytes 2023-12-05T22:39:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 64450560 bytes 2023-12-05T22:39:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 107996160 bytes 2023-12-05T22:39:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 126190592 bytes 2023-12-05T22:39:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 181986304 bytes 2023-12-05T22:39:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 203006976 bytes 2023-12-05T22:39:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 203614208 bytes 2023-12-05T22:39:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 272759808 bytes 2023-12-05T22:39:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 328146944 bytes 2023-12-05T22:39:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 363927552 bytes 2023-12-05T22:39:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 410015744 bytes 2023-12-05T22:39:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 429420544 bytes 2023-12-05T22:39:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 541125632 bytes 2023-12-05T22:39:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 574505984 bytes 2023-12-05T22:39:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 579812352 bytes 2023-12-05T22:39:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 586811392 bytes 2023-12-05T22:39:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 676727808 bytes 2023-12-05T22:39:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 723664896 bytes 2023-12-05T22:39:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 749063168 bytes 2023-12-05T22:39:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 786062336 bytes 2023-12-05T22:39:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 841236480 bytes 2023-12-05T22:39:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 842851328 bytes 2023-12-05T22:39:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 875730944 bytes 2023-12-05T22:39:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 978329600 bytes 2023-12-05T22:39:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 989535232 bytes 2023-12-05T22:39:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1024852992 bytes 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:39:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2023-12-05T22:39:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:39:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:39:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-05T22:39:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:39:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-05T22:39:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:39:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_43': 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 142ebbfd9975 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.142ebbfd9975 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:39:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:39 2023 2023-12-05T22:39:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:39 2023 2023-12-05T22:39:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:39 2023 2023-12-05T22:39:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:39:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:39:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: a0acf697-0cfe-4b8b-880e-ea4251b3ce8c 2023-12-05T22:39:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:39:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:39:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-05T22:39:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:39:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:39:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2023-12-05T22:39:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:39:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:39:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 47288320 bytes 2023-12-05T22:39:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 79489024 bytes 2023-12-05T22:39:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 129409024 bytes 2023-12-05T22:39:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 178727936 bytes 2023-12-05T22:39:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 221119488 bytes 2023-12-05T22:39:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 299656192 bytes 2023-12-05T22:39:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 325130240 bytes 2023-12-05T22:39:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 357476352 bytes 2023-12-05T22:39:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 358079488 bytes 2023-12-05T22:39:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 415642624 bytes 2023-12-05T22:39:42.357 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-05T22:39:42.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-05T22:39:42.358 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-05T22:39:42.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:39:42.362+0000 7f6b96a45640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 50153) UID: 1000 2023-12-05T22:39:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 499892224 bytes 2023-12-05T22:39:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 527893504 bytes 2023-12-05T22:39:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 570484736 bytes 2023-12-05T22:39:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 628009984 bytes 2023-12-05T22:39:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 673849344 bytes 2023-12-05T22:39:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 716166144 bytes 2023-12-05T22:39:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 790725632 bytes 2023-12-05T22:39:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 811411456 bytes 2023-12-05T22:39:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 812022784 bytes 2023-12-05T22:39:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 868086784 bytes 2023-12-05T22:39:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 943462400 bytes 2023-12-05T22:39:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 979430400 bytes 2023-12-05T22:39:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1028921344 bytes 2023-12-05T22:39:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1057906688 bytes 2023-12-05T22:39:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1094303744 bytes 2023-12-05T22:39:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1094917120 bytes 2023-12-05T22:39:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1185864704 bytes 2023-12-05T22:39:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1189299200 bytes 2023-12-05T22:39:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1190743040 bytes 2023-12-05T22:39:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1212860416 bytes 2023-12-05T22:39:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1248633856 bytes 2023-12-05T22:39:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1288374272 bytes 2023-12-05T22:39:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1327481856 bytes 2023-12-05T22:39:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1360258048 bytes 2023-12-05T22:39:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1441482752 bytes 2023-12-05T22:39:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1483089920 bytes 2023-12-05T22:39:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1545204736 bytes 2023-12-05T22:39:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1589135360 bytes 2023-12-05T22:39:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1647128576 bytes 2023-12-05T22:39:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1673674752 bytes 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:39:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:39:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:39:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2023-12-05T22:39:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:39:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:39:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:39:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-05T22:39:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:39:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:39:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-05T22:39:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:39:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_44': 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14362c7600e5 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14362c7600e5 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:48 2023 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:48 2023 2023-12-05T22:39:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:48 2023 2023-12-05T22:39:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:39:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:39:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c768d0f5-279d-48dd-8b01-214e09ad3ae9 2023-12-05T22:39:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-05T22:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:39:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:39:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:39:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:39:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:39:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 25747456 bytes 2023-12-05T22:39:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 68793344 bytes 2023-12-05T22:39:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 73410560 bytes 2023-12-05T22:39:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 105866240 bytes 2023-12-05T22:39:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 135995392 bytes 2023-12-05T22:39:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 186683392 bytes 2023-12-05T22:39:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 244293632 bytes 2023-12-05T22:39:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 274619392 bytes 2023-12-05T22:39:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 354192384 bytes 2023-12-05T22:39:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 401857536 bytes 2023-12-05T22:39:51.358 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-05T22:39:51.359 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:39:51.359 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-05T22:39:51.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-05T22:39:51.359 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-05T22:39:51.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-05T22:39:51.362 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-05T22:39:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 493516800 bytes 2023-12-05T22:39:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 532269056 bytes 2023-12-05T22:39:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 568346624 bytes 2023-12-05T22:39:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 588715008 bytes 2023-12-05T22:39:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 623082496 bytes 2023-12-05T22:39:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 699154432 bytes 2023-12-05T22:39:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 711643136 bytes 2023-12-05T22:39:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 722970624 bytes 2023-12-05T22:39:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 734783488 bytes 2023-12-05T22:39:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 777122816 bytes 2023-12-05T22:39:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 789427200 bytes 2023-12-05T22:39:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 822160384 bytes 2023-12-05T22:39:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 847900672 bytes 2023-12-05T22:39:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 858489856 bytes 2023-12-05T22:39:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 876598272 bytes 2023-12-05T22:39:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 955142144 bytes 2023-12-05T22:39:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 990273536 bytes 2023-12-05T22:39:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1016815616 bytes 2023-12-05T22:39:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1022794752 bytes 2023-12-05T22:39:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1048901632 bytes 2023-12-05T22:39:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1071440896 bytes 2023-12-05T22:39:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1133262848 bytes 2023-12-05T22:39:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1183757312 bytes 2023-12-05T22:39:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1196267520 bytes 2023-12-05T22:39:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1257077760 bytes 2023-12-05T22:39:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1295923200 bytes 2023-12-05T22:39:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1344703488 bytes 2023-12-05T22:39:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1366798336 bytes 2023-12-05T22:39:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1443817472 bytes 2023-12-05T22:39:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1486731264 bytes 2023-12-05T22:39:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1525381120 bytes 2023-12-05T22:39:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1555252224 bytes 2023-12-05T22:39:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1593861120 bytes 2023-12-05T22:39:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1600713728 bytes 2023-12-05T22:39:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1642060800 bytes 2023-12-05T22:39:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1674610688 bytes 2023-12-05T22:39:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1698171904 bytes 2023-12-05T22:39:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1765920768 bytes 2023-12-05T22:39:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1813996544 bytes 2023-12-05T22:39:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1825214464 bytes 2023-12-05T22:39:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1859380224 bytes 2023-12-05T22:39:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1883981824 bytes 2023-12-05T22:39:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1971340288 bytes 2023-12-05T22:39:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2013577216 bytes 2023-12-05T22:39:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2060026880 bytes 2023-12-05T22:39:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2128933888 bytes 2023-12-05T22:40:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2157199360 bytes 2023-12-05T22:40:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2193522688 bytes 2023-12-05T22:40:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2203945984 bytes 2023-12-05T22:40:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2204549120 bytes 2023-12-05T22:40:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2217640960 bytes 2023-12-05T22:40:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2272998400 bytes 2023-12-05T22:40:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2312429568 bytes 2023-12-05T22:40:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2343402496 bytes 2023-12-05T22:40:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2399163392 bytes 2023-12-05T22:40:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2447605760 bytes 2023-12-05T22:40:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2468637696 bytes 2023-12-05T22:40:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2490930176 bytes 2023-12-05T22:40:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2543928320 bytes 2023-12-05T22:40:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2023-12-05T22:40:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:40:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:40:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-05T22:40:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:40:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-05T22:40:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:40:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-05T22:40:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_45': 2023-12-05T22:40:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:40:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:40:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:40:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14441c696a31 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14441c696a31 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:02 2023 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:02 2023 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:02 2023 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1f88b449-0d2e-442c-b595-3a88ba6e5d1a 2023-12-05T22:40:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:40:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:40:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-05T22:40:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:40:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2023-12-05T22:40:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:40:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:40:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 8102912 bytes 2023-12-05T22:40:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 71407616 bytes 2023-12-05T22:40:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 146343936 bytes 2023-12-05T22:40:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 150108160 bytes 2023-12-05T22:40:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 190267392 bytes 2023-12-05T22:40:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 190894080 bytes 2023-12-05T22:40:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 235550720 bytes 2023-12-05T22:40:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 252254208 bytes 2023-12-05T22:40:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 281017344 bytes 2023-12-05T22:40:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 309815296 bytes 2023-12-05T22:40:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 348663808 bytes 2023-12-05T22:40:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 414930944 bytes 2023-12-05T22:40:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 484546560 bytes 2023-12-05T22:40:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 515477504 bytes 2023-12-05T22:40:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 551971840 bytes 2023-12-05T22:40:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 576440320 bytes 2023-12-05T22:40:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 631072768 bytes 2023-12-05T22:40:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 715223040 bytes 2023-12-05T22:40:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 749199360 bytes 2023-12-05T22:40:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 852400128 bytes 2023-12-05T22:40:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 920650752 bytes 2023-12-05T22:40:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1028657152 bytes 2023-12-05T22:40:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1067042816 bytes 2023-12-05T22:40:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1089199104 bytes 2023-12-05T22:40:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1096192000 bytes 2023-12-05T22:40:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1122215936 bytes 2023-12-05T22:40:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1128835072 bytes 2023-12-05T22:40:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1130238976 bytes 2023-12-05T22:40:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1150077952 bytes 2023-12-05T22:40:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1156647936 bytes 2023-12-05T22:40:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1222711296 bytes 2023-12-05T22:40:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1291088896 bytes 2023-12-05T22:40:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1309206528 bytes 2023-12-05T22:40:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1359354880 bytes 2023-12-05T22:40:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1417862144 bytes 2023-12-05T22:40:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1455821824 bytes 2023-12-05T22:40:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1520835584 bytes 2023-12-05T22:40:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1528565760 bytes 2023-12-05T22:40:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1567304704 bytes 2023-12-05T22:40:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1608284160 bytes 2023-12-05T22:40:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1630086144 bytes 2023-12-05T22:40:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1630717952 bytes 2023-12-05T22:40:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1673377792 bytes 2023-12-05T22:40:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1713053696 bytes 2023-12-05T22:40:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1746527232 bytes 2023-12-05T22:40:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1791899648 bytes 2023-12-05T22:40:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1792507904 bytes 2023-12-05T22:40:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1797784576 bytes 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:40:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:40:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:40:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:40:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2023-12-05T22:40:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-05T22:40:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:40:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_46': 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 144ce706ba85 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.144ce706ba85 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:40:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:40:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:12 2023 2023-12-05T22:40:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:12 2023 2023-12-05T22:40:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:12 2023 2023-12-05T22:40:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:40:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:40:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4dff6617-95c0-4b5e-9363-8846654b6996 2023-12-05T22:40:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:40:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:40:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-05T22:40:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:40:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:40:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:40:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:40:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:40:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 20899840 bytes 2023-12-05T22:40:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 104996864 bytes 2023-12-05T22:40:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 126427136 bytes 2023-12-05T22:40:13.846 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:40:13.846+0000 7f1618813640 -1 reset not still connected to 0x558e23457040 2023-12-05T22:40:13.848 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:40:13.846+0000 7f1618813640 -1 reset not still connected to 0x558e24fdadd0 2023-12-05T22:40:13.851 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:40:13.850+0000 7f1618813640 -1 reset not still connected to 0x558e250e52b0 2023-12-05T22:40:13.851 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:40:13.850+0000 7f1618813640 -1 reset not still connected to 0x558e250e55f0 2023-12-05T22:40:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 209827840 bytes 2023-12-05T22:40:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 295428096 bytes 2023-12-05T22:40:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 321339392 bytes 2023-12-05T22:40:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 408650752 bytes 2023-12-05T22:40:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 463894528 bytes 2023-12-05T22:40:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 509250560 bytes 2023-12-05T22:40:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 545555456 bytes 2023-12-05T22:40:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 622031872 bytes 2023-12-05T22:40:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 737972224 bytes 2023-12-05T22:40:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 793290752 bytes 2023-12-05T22:40:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 866904064 bytes 2023-12-05T22:40:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 900823040 bytes 2023-12-05T22:40:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 922661888 bytes 2023-12-05T22:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 946685952 bytes 2023-12-05T22:40:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 994284544 bytes 2023-12-05T22:40:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1031503872 bytes 2023-12-05T22:40:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1103705088 bytes 2023-12-05T22:40:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1139613696 bytes 2023-12-05T22:40:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1142320128 bytes 2023-12-05T22:40:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1203357696 bytes 2023-12-05T22:40:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1226172416 bytes 2023-12-05T22:40:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1264479232 bytes 2023-12-05T22:40:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1272884224 bytes 2023-12-05T22:40:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1348231168 bytes 2023-12-05T22:40:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1391428608 bytes 2023-12-05T22:40:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1452101632 bytes 2023-12-05T22:40:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1473845248 bytes 2023-12-05T22:40:19.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.32317312756428085) > weight (0.25) 2023-12-05T22:40:19.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5886099309038487) > weight (0.25) 2023-12-05T22:40:19.461 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-05T22:40:19.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-05T22:40:19.462 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-05T22:40:19.462 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-05T22:40:19.462 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:40:19.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:40:19.472+0000 7f2ac0514640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 51596) UID: 1000 2023-12-05T22:40:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1474448384 bytes 2023-12-05T22:40:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1487022080 bytes 2023-12-05T22:40:19.770 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:40:19.776+0000 7fdd570d4640 -1 reset not still connected to 0x559d27730c30 2023-12-05T22:40:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1501028352 bytes 2023-12-05T22:40:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1579031552 bytes 2023-12-05T22:40:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1621249024 bytes 2023-12-05T22:40:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1681097728 bytes 2023-12-05T22:40:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1697574912 bytes 2023-12-05T22:40:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1764878336 bytes 2023-12-05T22:40:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1819057152 bytes 2023-12-05T22:40:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1843130368 bytes 2023-12-05T22:40:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1880748032 bytes 2023-12-05T22:40:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1948853248 bytes 2023-12-05T22:40:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1979653120 bytes 2023-12-05T22:40:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1981332480 bytes 2023-12-05T22:40:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1982345216 bytes 2023-12-05T22:40:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2043097088 bytes 2023-12-05T22:40:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2138527744 bytes 2023-12-05T22:40:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2158898176 bytes 2023-12-05T22:40:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2178471936 bytes 2023-12-05T22:40:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2240078848 bytes 2023-12-05T22:40:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2278338560 bytes 2023-12-05T22:40:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2287448064 bytes 2023-12-05T22:40:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2321724416 bytes 2023-12-05T22:40:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2420041728 bytes 2023-12-05T22:40:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2484631552 bytes 2023-12-05T22:40:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2500747264 bytes 2023-12-05T22:40:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2566428672 bytes 2023-12-05T22:40:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2587609088 bytes 2023-12-05T22:40:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2590417920 bytes 2023-12-05T22:40:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2640357376 bytes 2023-12-05T22:40:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2715789312 bytes 2023-12-05T22:40:25.564 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-05T22:40:25.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-05T22:40:25.565 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-05T22:40:25.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-05T22:40:25.567 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-12-05T22:40:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2752062464 bytes 2023-12-05T22:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2827736064 bytes 2023-12-05T22:40:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2873903104 bytes 2023-12-05T22:40:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2917766144 bytes 2023-12-05T22:40:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2918380544 bytes 2023-12-05T22:40:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2969984000 bytes 2023-12-05T22:40:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3005595648 bytes 2023-12-05T22:40:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3054914560 bytes 2023-12-05T22:40:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3058823168 bytes 2023-12-05T22:40:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2023-12-05T22:40:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:40:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-05T22:40:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:40:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-05T22:40:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_47': 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 145fccc7aa74 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.145fccc7aa74 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:27 2023 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:27 2023 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:27 2023 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:40:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b3e4e6e0-3842-4e3a-8b00-addba2ae7c6f 2023-12-05T22:40:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:40:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:40:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-05T22:40:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:40:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:40:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2023-12-05T22:40:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:40:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:40:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 51567616 bytes 2023-12-05T22:40:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 96328704 bytes 2023-12-05T22:40:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 132603904 bytes 2023-12-05T22:40:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 159684608 bytes 2023-12-05T22:40:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 201164800 bytes 2023-12-05T22:40:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 258046976 bytes 2023-12-05T22:40:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 318605312 bytes 2023-12-05T22:40:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 409211904 bytes 2023-12-05T22:40:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 499540992 bytes 2023-12-05T22:40:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 545103872 bytes 2023-12-05T22:40:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 583415808 bytes 2023-12-05T22:40:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 663006208 bytes 2023-12-05T22:40:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 699969536 bytes 2023-12-05T22:40:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 700580864 bytes 2023-12-05T22:40:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 720508928 bytes 2023-12-05T22:40:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 784653312 bytes 2023-12-05T22:40:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 828499968 bytes 2023-12-05T22:40:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 929471488 bytes 2023-12-05T22:40:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 954369024 bytes 2023-12-05T22:40:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 984883200 bytes 2023-12-05T22:40:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1037758464 bytes 2023-12-05T22:40:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1088825344 bytes 2023-12-05T22:40:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1122067456 bytes 2023-12-05T22:40:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1151013888 bytes 2023-12-05T22:40:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1199693824 bytes 2023-12-05T22:40:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1259022336 bytes 2023-12-05T22:40:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1278473216 bytes 2023-12-05T22:40:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1350903808 bytes 2023-12-05T22:40:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1376109568 bytes 2023-12-05T22:40:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1408051200 bytes 2023-12-05T22:40:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1418899456 bytes 2023-12-05T22:40:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1424740352 bytes 2023-12-05T22:40:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1481397248 bytes 2023-12-05T22:40:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1515806720 bytes 2023-12-05T22:40:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1534656512 bytes 2023-12-05T22:40:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1622936576 bytes 2023-12-05T22:40:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1696058368 bytes 2023-12-05T22:40:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1751897088 bytes 2023-12-05T22:40:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1752515584 bytes 2023-12-05T22:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:40:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-05T22:40:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:40:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-05T22:40:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_48': 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14681961d661 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14681961d661 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:35 2023 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:35 2023 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:35 2023 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:40:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:40:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4c38d01c-4d6b-4116-a8ce-10e93c8bf38b 2023-12-05T22:40:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:40:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-05T22:40:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:40:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:40:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:40:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:40:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:40:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 39683072 bytes 2023-12-05T22:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 56223744 bytes 2023-12-05T22:40:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 106027008 bytes 2023-12-05T22:40:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 135664640 bytes 2023-12-05T22:40:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 177250304 bytes 2023-12-05T22:40:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 199665664 bytes 2023-12-05T22:40:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 216259584 bytes 2023-12-05T22:40:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 263383040 bytes 2023-12-05T22:40:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 378936320 bytes 2023-12-05T22:40:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 436844544 bytes 2023-12-05T22:40:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 463709184 bytes 2023-12-05T22:40:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 530272256 bytes 2023-12-05T22:40:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 573190144 bytes 2023-12-05T22:40:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 573787136 bytes 2023-12-05T22:40:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 637749248 bytes 2023-12-05T22:40:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 668803072 bytes 2023-12-05T22:40:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 706313216 bytes 2023-12-05T22:40:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 748028928 bytes 2023-12-05T22:40:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 826935296 bytes 2023-12-05T22:40:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 870913024 bytes 2023-12-05T22:40:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 907828224 bytes 2023-12-05T22:40:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 954446848 bytes 2023-12-05T22:40:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1037455360 bytes 2023-12-05T22:40:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1093394432 bytes 2023-12-05T22:40:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1122714624 bytes 2023-12-05T22:40:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1207797760 bytes 2023-12-05T22:40:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1208417280 bytes 2023-12-05T22:40:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1259009024 bytes 2023-12-05T22:40:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1266854912 bytes 2023-12-05T22:40:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1300000768 bytes 2023-12-05T22:40:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1342236672 bytes 2023-12-05T22:40:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1360834560 bytes 2023-12-05T22:40:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1391429632 bytes 2023-12-05T22:40:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1399683072 bytes 2023-12-05T22:40:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1400313856 bytes 2023-12-05T22:40:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1438235648 bytes 2023-12-05T22:40:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1558022144 bytes 2023-12-05T22:40:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1578786816 bytes 2023-12-05T22:40:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1580107776 bytes 2023-12-05T22:40:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1633294336 bytes 2023-12-05T22:40:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1706278912 bytes 2023-12-05T22:40:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1726695424 bytes 2023-12-05T22:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1752089600 bytes 2023-12-05T22:40:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1786414080 bytes 2023-12-05T22:40:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1817282560 bytes 2023-12-05T22:40:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1843085312 bytes 2023-12-05T22:40:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1889914880 bytes 2023-12-05T22:40:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1936947200 bytes 2023-12-05T22:40:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1972646912 bytes 2023-12-05T22:40:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2022824960 bytes 2023-12-05T22:40:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2056605696 bytes 2023-12-05T22:40:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2059342848 bytes 2023-12-05T22:40:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2109197312 bytes 2023-12-05T22:40:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2166642688 bytes 2023-12-05T22:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2174226432 bytes 2023-12-05T22:40:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2186780672 bytes 2023-12-05T22:40:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2187390976 bytes 2023-12-05T22:40:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2261823488 bytes 2023-12-05T22:40:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2310805504 bytes 2023-12-05T22:40:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2382826496 bytes 2023-12-05T22:40:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2424716288 bytes 2023-12-05T22:40:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2436694016 bytes 2023-12-05T22:40:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2441965568 bytes 2023-12-05T22:40:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2449547264 bytes 2023-12-05T22:40:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2503106560 bytes 2023-12-05T22:40:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2516842496 bytes 2023-12-05T22:40:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2545272832 bytes 2023-12-05T22:40:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2581535744 bytes 2023-12-05T22:40:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:40:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:40:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:40:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:40:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:40:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:40:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:40:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-05T22:40:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:40:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_49': 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1471f8d49f33 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1471f8d49f33 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:49 2023 2023-12-05T22:40:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:49 2023 2023-12-05T22:40:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:49 2023 2023-12-05T22:40:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:40:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:40:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f77f683c-79d4-42df-af67-46a03922b77d 2023-12-05T22:40:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:40:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:40:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-05T22:40:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:40:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:40:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2023-12-05T22:40:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:40:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:40:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 46867456 bytes 2023-12-05T22:40:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 47485952 bytes 2023-12-05T22:40:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 56799232 bytes 2023-12-05T22:40:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 126721024 bytes 2023-12-05T22:40:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 135025664 bytes 2023-12-05T22:40:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 229199872 bytes 2023-12-05T22:40:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 245786624 bytes 2023-12-05T22:40:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 286295040 bytes 2023-12-05T22:40:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 338484224 bytes 2023-12-05T22:40:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 359245824 bytes 2023-12-05T22:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 405464064 bytes 2023-12-05T22:40:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 416153600 bytes 2023-12-05T22:40:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 452894720 bytes 2023-12-05T22:40:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 494952448 bytes 2023-12-05T22:40:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 534582272 bytes 2023-12-05T22:40:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 559223808 bytes 2023-12-05T22:40:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 633083904 bytes 2023-12-05T22:40:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 670800896 bytes 2023-12-05T22:40:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 699676672 bytes 2023-12-05T22:40:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 828361728 bytes 2023-12-05T22:40:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 868722688 bytes 2023-12-05T22:40:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 934821888 bytes 2023-12-05T22:40:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1004502016 bytes 2023-12-05T22:40:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1045145600 bytes 2023-12-05T22:40:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1077655552 bytes 2023-12-05T22:40:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1106460672 bytes 2023-12-05T22:40:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1111663616 bytes 2023-12-05T22:40:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1127036928 bytes 2023-12-05T22:40:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1191583744 bytes 2023-12-05T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1258368000 bytes 2023-12-05T22:40:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1338343424 bytes 2023-12-05T22:40:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1382067200 bytes 2023-12-05T22:40:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1435431936 bytes 2023-12-05T22:40:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1460261888 bytes 2023-12-05T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1486154752 bytes 2023-12-05T22:40:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:40:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:40:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:40:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n snapshot ] 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 128M ] 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2023-12-05T22:40:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ is_pool_mirror_mode_image mirror 2023-12-05T22:40:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ image = image ] 2023-12-05T22:40:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:40:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-05T22:40:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:40:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:40:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-05T22:40:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:40:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-05T22:40:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_50': 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 147aa06d6011 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.147aa06d6011 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:57 2023 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:57 2023 2023-12-05T22:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:57 2023 2023-12-05T22:40:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:40:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:40:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: dab6d77c-da14-4bca-b431-d1248425b922 2023-12-05T22:40:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:40:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n ] 2023-12-05T22:40:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-05T22:40:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:40:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:40:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:40:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:40:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2023-12-05T22:40:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:40:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:40:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 38141952 bytes 2023-12-05T22:40:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 66740224 bytes 2023-12-05T22:40:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 147084288 bytes 2023-12-05T22:40:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 236508160 bytes 2023-12-05T22:40:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 271562752 bytes 2023-12-05T22:40:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 292422656 bytes 2023-12-05T22:41:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 404953088 bytes 2023-12-05T22:41:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 460551168 bytes 2023-12-05T22:41:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 545183744 bytes 2023-12-05T22:41:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 571350016 bytes 2023-12-05T22:41:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 613929984 bytes 2023-12-05T22:41:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 640039936 bytes 2023-12-05T22:41:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 667935744 bytes 2023-12-05T22:41:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 683241472 bytes 2023-12-05T22:41:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 696154112 bytes 2023-12-05T22:41:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 750274560 bytes 2023-12-05T22:41:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 783823872 bytes 2023-12-05T22:41:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 861271040 bytes 2023-12-05T22:41:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 891193344 bytes 2023-12-05T22:41:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 942610432 bytes 2023-12-05T22:41:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 961217536 bytes 2023-12-05T22:41:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 993241088 bytes 2023-12-05T22:41:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1050802176 bytes 2023-12-05T22:41:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1080932352 bytes 2023-12-05T22:41:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1186935808 bytes 2023-12-05T22:41:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1223208960 bytes 2023-12-05T22:41:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1268657152 bytes 2023-12-05T22:41:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1288087552 bytes 2023-12-05T22:41:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1332078592 bytes 2023-12-05T22:41:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1403627520 bytes 2023-12-05T22:41:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1444354048 bytes 2023-12-05T22:41:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1480848384 bytes 2023-12-05T22:41:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1539741696 bytes 2023-12-05T22:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1593353216 bytes 2023-12-05T22:41:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1643267072 bytes 2023-12-05T22:41:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1699057664 bytes 2023-12-05T22:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1779386368 bytes 2023-12-05T22:41:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1809137664 bytes 2023-12-05T22:41:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1871365120 bytes 2023-12-05T22:41:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1882096640 bytes 2023-12-05T22:41:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1947644928 bytes 2023-12-05T22:41:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1999065088 bytes 2023-12-05T22:41:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2019574784 bytes 2023-12-05T22:41:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:41:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:41:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:41:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_count=50 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local count 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_count=0 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 40 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep images: 2023-12-05T22:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cut -d -f 2 2023-12-05T22:41:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:41:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count=50 2023-12-05T22:41:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 50 = 50 2023-12-05T22:41:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 40 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep image health: 2023-12-05T22:41:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cut -d -f 3 2023-12-05T22:41:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:41:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=OK 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = ERROR 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = OK 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_count=50 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local count 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_count=0 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 40 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2023-12-05T22:41:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-05T22:41:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep images: 2023-12-05T22:41:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cut -d -f 2 2023-12-05T22:41:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:41:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count=50 2023-12-05T22:41:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 50 = 50 2023-12-05T22:41:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-05T22:41:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:41:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state 2023-12-05T22:41:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 40 2023-12-05T22:41:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2023-12-05T22:41:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + rbd --cluster cluster1 -p mirror mirror pool status 2023-12-05T22:41:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep image health: 2023-12-05T22:41:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cut -d -f 3 2023-12-05T22:41:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:41:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=WARNING 2023-12-05T22:41:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test WARNING = ERROR 2023-12-05T22:41:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test WARNING = OK 2023-12-05T22:41:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 2 -ne 1 2023-12-05T22:41:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 30 2023-12-05T22:41:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-05T22:41:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep image health: 2023-12-05T22:41:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cut -d -f 3 2023-12-05T22:41:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:41:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=OK 2023-12-05T22:41:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = ERROR 2023-12-05T22:41:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = OK 2023-12-05T22:41:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog TEST: compare many images 2023-12-05T22:41:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.yK28jjcyo4/rbd-mirror.test.log 2023-12-05T22:41:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ date +%F %T 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2023-12-05 22:41:37 TEST: compare many images 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2023-12-05 22:41:37 TEST: compare many images 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 50 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:41:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:41:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:41:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:41:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:41:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:41:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:41:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:41:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:41:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-05T22:41:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:41:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-05T22:41:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:41:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:41:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:41:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:41:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:41:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:41:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-05T22:41:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:41:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:41:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:41:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-05T22:41:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:41:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_1.status 2023-12-05T22:41:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_1.status 2023-12-05T22:41:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:41:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.status 2023-12-05T22:41:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_1.status 2023-12-05T22:41:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_1.status 2023-12-05T22:41:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:41:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-05T22:41:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-05T22:41:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 119 2023-12-05T22:41:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.status 2023-12-05T22:41:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.status 2023-12-05T22:41:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-05T22:41:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:41:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:41:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=119 2023-12-05T22:41:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:41:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:41:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-05T22:41:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:41:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:41:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=68 2023-12-05T22:41:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = 68 2023-12-05T22:41:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:41:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-05T22:41:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:41:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:41:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:41:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:41:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:41:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:41:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:41:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = 2023-12-05T22:41:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-05T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:41:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:41:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:41:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:41:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:41:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:41:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:41:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = 2023-12-05T22:41:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:41:43.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4661265538546576) > weight (0.25) 2023-12-05T22:41:43.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5093267082305689) > weight (0.25) 2023-12-05T22:41:43.363 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-05T22:41:43.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-05T22:41:43.364 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-05T22:41:43.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:41:43.373+0000 7f0dae481640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 52007) UID: 1000 2023-12-05T22:41:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.status 2023-12-05T22:41:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-05T22:41:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:41:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:41:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=119 2023-12-05T22:41:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = 119 2023-12-05T22:41:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-05T22:41:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:41:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:41:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:41:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-05T22:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_1': 2023-12-05T22:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c540ce6f4d 2023-12-05T22:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c540ce6f4d 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:32:25 2023 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:32:25 2023 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:32:25 2023 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6dc8b84c-1b31-4882-988d-5f4b4f42faf2 2023-12-05T22:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:41:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-05T22:41:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:41:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:41:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:41:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.export 2023-12-05T22:41:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.export 2023-12-05T22:41:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.export 2023-12-05T22:41:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.export 2023-12-05T22:41:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:41:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.export 2023-12-05T22:41:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:41:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.export 2023-12-05T22:41:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_1.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_1.export 2023-12-05T22:41:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:41:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2023-12-05T22:41:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-05T22:41:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:41:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:41:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:41:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:41:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:41:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:41:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:41:49.364 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-05T22:41:49.365 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:41:49.365 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-05T22:41:49.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-05T22:41:49.366 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-05T22:41:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-05T22:41:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:41:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:41:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:41:49.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-05T22:41:49.369 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:41:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-05T22:41:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2023-12-05T22:41:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2023-12-05T22:41:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:41:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-05T22:41:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 4 closest matches: 2023-12-05T22:41:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2023-12-05T22:41:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2023-12-05T22:41:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_28 2023-12-05T22:41:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2023-12-05T22:41:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-05T22:41:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:41:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:41:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-05T22:41:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-12-05T22:41:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:41:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:41:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:49.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:49.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-12-05T22:41:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 4 closest matches: 2023-12-05T22:41:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2023-12-05T22:41:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_22 2023-12-05T22:41:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_26 2023-12-05T22:41:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_27 2023-12-05T22:41:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:41:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:41:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:41:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-05T22:41:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2023-12-05T22:41:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2023-12-05T22:41:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:41:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-05T22:41:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 4 closest matches: 2023-12-05T22:41:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2023-12-05T22:41:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2023-12-05T22:41:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_28 2023-12-05T22:41:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2023-12-05T22:41:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-05T22:41:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:41:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:41:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:41:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-05T22:41:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-12-05T22:41:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:41:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-12-05T22:41:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:41:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:41:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:41:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:41:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-12-05T22:41:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 4 closest matches: 2023-12-05T22:41:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2023-12-05T22:41:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_22 2023-12-05T22:41:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_26 2023-12-05T22:41:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_27 2023-12-05T22:41:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:41:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:41:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:41:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-05T22:41:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2023-12-05T22:41:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2023-12-05T22:41:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:41:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-05T22:41:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:41:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_28 2023-12-05T22:41:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2023-12-05T22:41:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-05T22:41:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:41:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:41:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:41:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:41:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-05T22:41:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-12-05T22:41:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:41:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-12-05T22:41:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:41:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-12-05T22:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2023-12-05T22:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_27 2023-12-05T22:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:41:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:41:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:41:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:41:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-05T22:41:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2023-12-05T22:41:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2023-12-05T22:41:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-05T22:41:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:41:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:41:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:41:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:41:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:41:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-05T22:41:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:41:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_28 2023-12-05T22:41:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2023-12-05T22:41:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-05T22:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:41:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:41:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:41:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:41:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:41:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-05T22:41:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 4 closest matches: 2023-12-05T22:41:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2023-12-05T22:41:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_22 2023-12-05T22:41:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2023-12-05T22:41:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_26 2023-12-05T22:41:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:41:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:41:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:41:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:41:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:41:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-12-05T22:41:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:41:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2023-12-05T22:41:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_27 2023-12-05T22:41:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:41:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:42:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:42:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:42:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-05T22:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2023-12-05T22:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2023-12-05T22:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:42:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-05T22:42:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:42:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_28 2023-12-05T22:42:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2023-12-05T22:42:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:42:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:42:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-05T22:42:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 4 closest matches: 2023-12-05T22:42:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2023-12-05T22:42:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_22 2023-12-05T22:42:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2023-12-05T22:42:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_26 2023-12-05T22:42:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:42:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:42:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:42:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:42:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:42:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:42:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:42:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:42:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:42:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-12-05T22:42:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:42:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2023-12-05T22:42:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_27 2023-12-05T22:42:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:42:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:42:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:42:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-05T22:42:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:42:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:42:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:42:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-05T22:42:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2023-12-05T22:42:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2023-12-05T22:42:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-05T22:42:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:42:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:42:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:42:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-05T22:42:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:42:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_28 2023-12-05T22:42:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2023-12-05T22:42:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-05T22:42:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:42:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:42:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:42:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-05T22:42:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:42:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grep -i Replaying 2023-12-05T22:42:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo "state": "Replaying" 2023-12-05T22:42:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:42:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-05T22:42:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_2.status 2023-12-05T22:42:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_2.status 2023-12-05T22:42:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.status 2023-12-05T22:42:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_2.status 2023-12-05T22:42:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_2.status 2023-12-05T22:42:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-05T22:42:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-05T22:42:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 121 2023-12-05T22:42:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.status 2023-12-05T22:42:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.status 2023-12-05T22:42:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-05T22:42:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:42:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=121 2023-12-05T22:42:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:42:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-05T22:42:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=69 2023-12-05T22:42:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = 69 2023-12-05T22:42:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:42:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-05T22:42:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = 2023-12-05T22:42:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:42:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-05T22:42:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = 2023-12-05T22:42:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:42:19.568 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3120437218040554) > weight (0.25) 2023-12-05T22:42:19.569 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.49380416604769806) > weight (0.25) 2023-12-05T22:42:19.569 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5635684354483763) > weight (0.25) 2023-12-05T22:42:19.569 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6378566459302387) > weight (0.25) 2023-12-05T22:42:19.569 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-05T22:42:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.status 2023-12-05T22:42:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-05T22:42:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=121 2023-12-05T22:42:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = 121 2023-12-05T22:42:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-05T22:42:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-05T22:42:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_2': 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1140b161628b 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1140b161628b 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:32:37 2023 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:32:37 2023 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:32:37 2023 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9b0c0c99-2835-43a4-88c9-c3397c0e79b3 2023-12-05T22:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-05T22:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.export 2023-12-05T22:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.export 2023-12-05T22:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.export 2023-12-05T22:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.export 2023-12-05T22:42:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.export 2023-12-05T22:42:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.export 2023-12-05T22:42:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_2.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_2.export 2023-12-05T22:42:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2023-12-05T22:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-05T22:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:42:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-05T22:42:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:42:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:42:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:42:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-05T22:42:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:42:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:42:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:42:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:42:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:42:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-05T22:42:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2023-12-05T22:42:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_39 2023-12-05T22:42:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:42:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-12-05T22:42:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:42:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:42:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_3.status 2023-12-05T22:42:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_3.status 2023-12-05T22:42:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.status 2023-12-05T22:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_3.status 2023-12-05T22:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_3.status 2023-12-05T22:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-05T22:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-05T22:42:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 123 2023-12-05T22:42:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.status 2023-12-05T22:42:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.status 2023-12-05T22:42:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-05T22:42:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:42:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=123 2023-12-05T22:42:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:42:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-05T22:42:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=70 2023-12-05T22:42:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = 70 2023-12-05T22:42:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:42:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-05T22:42:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = 2023-12-05T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:42:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-05T22:42:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = 2023-12-05T22:42:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:42:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.status 2023-12-05T22:42:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-05T22:42:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=123 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = 123 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-05T22:42:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_3': 2023-12-05T22:42:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:42:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:42:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:42:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12252b0b54da 2023-12-05T22:42:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12252b0b54da 2023-12-05T22:42:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:42:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:32:50 2023 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:32:50 2023 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:32:50 2023 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:42:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 09ba1f5d-4c90-48e3-a545-11284c264b5a 2023-12-05T22:42:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:42:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-05T22:42:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:42:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:42:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.export 2023-12-05T22:42:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.export 2023-12-05T22:42:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.export 2023-12-05T22:42:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.export 2023-12-05T22:42:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.export 2023-12-05T22:42:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.export 2023-12-05T22:42:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_3.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_3.export 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:42:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-05T22:42:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-05T22:42:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:42:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-05T22:42:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ greplocal -i instance=cluster1 state 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:42:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:42:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:42:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:42:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-05T22:42:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:42:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:42:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-05T22:42:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2023-12-05T22:42:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_45 2023-12-05T22:42:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_48 2023-12-05T22:42:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-05T22:42:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:42:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:42:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:42:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:42:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:42:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-05T22:42:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:42:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:42:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:42:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_4.status 2023-12-05T22:42:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_4.status 2023-12-05T22:42:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.status 2023-12-05T22:42:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_4.status 2023-12-05T22:42:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_4.status 2023-12-05T22:42:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.status 2023-12-05T22:42:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-05T22:42:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-05T22:42:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 125 2023-12-05T22:42:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.status 2023-12-05T22:42:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.status 2023-12-05T22:42:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-05T22:42:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:42:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=125 2023-12-05T22:42:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.status 2023-12-05T22:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.status 2023-12-05T22:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-05T22:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 125 = 2023-12-05T22:42:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:42:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.status 2023-12-05T22:42:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.status 2023-12-05T22:42:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-05T22:42:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 125 = 2023-12-05T22:42:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:42:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.status 2023-12-05T22:42:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.status 2023-12-05T22:42:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-05T22:42:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=125 2023-12-05T22:42:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 125 = 125 2023-12-05T22:42:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-05T22:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_4': 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289a23942da 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289a23942da 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:42:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:03 2023 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:03 2023 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:03 2023 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: aa119937-ee52-43d0-81c6-311c63b77e8f 2023-12-05T22:42:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.export 2023-12-05T22:42:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.export 2023-12-05T22:42:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.export 2023-12-05T22:42:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.export 2023-12-05T22:42:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.export 2023-12-05T22:42:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.export 2023-12-05T22:42:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_4.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_4.export 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-05T22:42:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:42:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:42:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-05T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:42:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:42:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:42:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:42:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-05T22:42:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:42:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:42:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:42:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-05T22:42:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-05T22:42:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:42:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-05T22:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:42:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:42:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:42:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:42:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-05T22:42:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-05T22:42:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:42:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:42:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-12-05T22:42:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:42:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:42:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:42:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:42:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:42:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:42:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:42:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-12-05T22:42:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:42:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:42:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:42:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:42:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_5.status 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_5.status 2023-12-05T22:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.status 2023-12-05T22:42:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_5.status 2023-12-05T22:42:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo cluster1-mirror-image_5.status 2023-12-05T22:42:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -e s|[/ ]|_|g 2023-12-05T22:42:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.status 2023-12-05T22:42:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-05T22:42:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-05T22:42:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 127 2023-12-05T22:42:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.status 2023-12-05T22:42:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.status 2023-12-05T22:42:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-05T22:42:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:42:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=127 2023-12-05T22:42:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:42:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.status 2023-12-05T22:42:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.status 2023-12-05T22:42:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-05T22:42:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=72 2023-12-05T22:42:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 127 = 72 2023-12-05T22:42:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:42:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.status 2023-12-05T22:42:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.status 2023-12-05T22:42:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-05T22:42:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 127 = 2023-12-05T22:42:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:42:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.status 2023-12-05T22:42:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.status 2023-12-05T22:42:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-05T22:42:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=127 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 127 = 127 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-05T22:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_5': 2023-12-05T22:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c55a98a71d 2023-12-05T22:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c55a98a71d 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:18 2023 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:18 2023 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:18 2023 2023-12-05T22:42:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:42:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:42:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:42:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 23aac29b-db31-4b69-bed6-9ec5773f48c0 2023-12-05T22:42:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-05T22:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.export 2023-12-05T22:42:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.export 2023-12-05T22:42:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.export 2023-12-05T22:42:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.export 2023-12-05T22:42:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.export 2023-12-05T22:42:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.export 2023-12-05T22:42:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_5.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_5.export 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:42:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:42:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:42:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:42:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:42:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:42:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-05T22:42:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:42:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:42:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:42:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:42:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_6.status 2023-12-05T22:42:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_6.status 2023-12-05T22:42:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.status 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_6.status 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo cluster1-mirror-image_6.status 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -e s|[/ ]|_|g 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-05T22:42:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 129 2023-12-05T22:42:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.status 2023-12-05T22:42:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.status 2023-12-05T22:42:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-05T22:42:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:42:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=129 2023-12-05T22:42:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:42:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-05T22:42:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=73 2023-12-05T22:42:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = 73 2023-12-05T22:42:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:42:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-05T22:42:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = 2023-12-05T22:42:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-05T22:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = 2023-12-05T22:42:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:42:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.status 2023-12-05T22:42:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-05T22:42:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=129 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = 129 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-05T22:42:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_6': 2023-12-05T22:42:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:42:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:42:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:42:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1140d2442bf8 2023-12-05T22:42:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1140d2442bf8 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:26 2023 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:26 2023 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:26 2023 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ee1e6e37-e0d9-4a81-a633-71147ddff486 2023-12-05T22:42:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:42:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-05T22:42:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:42:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:42:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.export 2023-12-05T22:42:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.export 2023-12-05T22:42:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.export 2023-12-05T22:42:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.export 2023-12-05T22:42:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.export 2023-12-05T22:42:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:42:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.export 2023-12-05T22:42:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_6.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_6.export 2023-12-05T22:42:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2023-12-05T22:42:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-05T22:42:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:42:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:42:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-05T22:42:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-05T22:42:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:42:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:42:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:42:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-05T22:42:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:42:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:42:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:42:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-05T22:42:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:42:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:42:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:42:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:42:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:42:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:42:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:42:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-05T22:42:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:42:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:42:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:42:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:42:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:42:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:42:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:42:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-05T22:42:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:42:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:42:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:42:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:42:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:42:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:42:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:42:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-05T22:42:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_7.status 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_7.status 2023-12-05T22:42:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.status 2023-12-05T22:42:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_7.status 2023-12-05T22:42:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_7.status 2023-12-05T22:42:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:42:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:42:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-05T22:42:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-05T22:42:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 131 2023-12-05T22:42:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.status 2023-12-05T22:42:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:42:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.status 2023-12-05T22:42:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-05T22:42:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:42:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=131 2023-12-05T22:42:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:42:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:42:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:42:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-05T22:42:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = 2023-12-05T22:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:42:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:42:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:42:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-05T22:42:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = 2023-12-05T22:42:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:42:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:42:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:42:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:42:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:42:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:42:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-05T22:42:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:42:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:42:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = 2023-12-05T22:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:43:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:43:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:43:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.status 2023-12-05T22:43:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-05T22:43:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=131 2023-12-05T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = 131 2023-12-05T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-05T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:43:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-05T22:43:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_7': 2023-12-05T22:43:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1225ce7152cb 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1225ce7152cb 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:39 2023 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:39 2023 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:39 2023 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:43:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 01c4381c-2953-4889-8d37-61725def0e37 2023-12-05T22:43:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:43:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-05T22:43:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:43:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:43:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.export 2023-12-05T22:43:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.export 2023-12-05T22:43:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.export 2023-12-05T22:43:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.export 2023-12-05T22:43:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.export 2023-12-05T22:43:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.export 2023-12-05T22:43:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_7.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_7.export 2023-12-05T22:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2023-12-05T22:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-05T22:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:43:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:05.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:43:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-05T22:43:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:43:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:43:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:43:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:43:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:43:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:43:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-05T22:43:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_8.status 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_8.status 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.status 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_8.status 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_8.status 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-05T22:43:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 133 2023-12-05T22:43:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.status 2023-12-05T22:43:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.status 2023-12-05T22:43:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-05T22:43:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:43:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=133 2023-12-05T22:43:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:43:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-05T22:43:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=75 2023-12-05T22:43:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = 75 2023-12-05T22:43:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:43:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-05T22:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:43:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:43:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:43:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:43:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = 2023-12-05T22:43:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:43:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-05T22:43:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:43:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:43:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:43:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:43:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = 2023-12-05T22:43:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:43:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.status 2023-12-05T22:43:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-05T22:43:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=133 2023-12-05T22:43:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = 133 2023-12-05T22:43:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-05T22:43:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-05T22:43:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_8': 2023-12-05T22:43:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:43:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289baefd654 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289baefd654 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:47 2023 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:47 2023 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:47 2023 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:43:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 5e8d2584-d40c-4e8c-bda1-2d1342a10e52 2023-12-05T22:43:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.export 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.export 2023-12-05T22:43:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.export 2023-12-05T22:43:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.export 2023-12-05T22:43:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.export 2023-12-05T22:43:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.export 2023-12-05T22:43:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_8.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_8.export 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:43:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-05T22:43:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-05T22:43:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:43:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:43:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:43:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-05T22:43:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:43:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:43:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:43:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:43:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:43:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-05T22:43:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-05T22:43:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-12-05T22:43:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:43:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-12-05T22:43:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:43:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:43:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:43:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:43:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:43:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:43:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-05T22:43:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_9.status 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_9.status 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.status 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_9.status 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_9.status 2023-12-05T22:43:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-05T22:43:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-05T22:43:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 135 2023-12-05T22:43:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.status 2023-12-05T22:43:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.status 2023-12-05T22:43:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-05T22:43:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:43:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=135 2023-12-05T22:43:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:43:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-05T22:43:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=76 2023-12-05T22:43:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 135 = 76 2023-12-05T22:43:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:43:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-05T22:43:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:43:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:43:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:43:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:43:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 135 = 2023-12-05T22:43:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:43:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-05T22:43:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:43:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:43:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:43:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:43:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 135 = 2023-12-05T22:43:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:43:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.status 2023-12-05T22:43:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-05T22:43:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=135 2023-12-05T22:43:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 135 = 135 2023-12-05T22:43:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-05T22:43:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-05T22:43:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_9': 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129d97f16a70 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129d97f16a70 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:33:55 2023 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:33:55 2023 2023-12-05T22:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:33:55 2023 2023-12-05T22:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d0e14dbe-197f-4ab0-8cb5-c1acd3e96372 2023-12-05T22:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.export 2023-12-05T22:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.export 2023-12-05T22:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.export 2023-12-05T22:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.export 2023-12-05T22:43:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.export 2023-12-05T22:43:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.export 2023-12-05T22:43:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_9.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_9.export 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:43:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-05T22:43:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-05T22:43:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:43:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:43:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirrorgrep -i state status 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: mirror/image_10 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:43:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-05T22:43:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-05T22:43:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-05T22:43:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:43:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-05T22:43:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-05T22:43:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:43:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:43:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:43:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:43:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-12-05T22:43:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:43:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:43:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:43:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:43:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_10.status 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_10.status 2023-12-05T22:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.status 2023-12-05T22:43:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_10.status 2023-12-05T22:43:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_10.status 2023-12-05T22:43:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.status 2023-12-05T22:43:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-05T22:43:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-05T22:43:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 137 2023-12-05T22:43:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.status 2023-12-05T22:43:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.status 2023-12-05T22:43:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-05T22:43:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=137 2023-12-05T22:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.status 2023-12-05T22:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.status 2023-12-05T22:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-05T22:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 137 = 2023-12-05T22:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:43:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.status 2023-12-05T22:43:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.status 2023-12-05T22:43:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-05T22:43:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:43:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:43:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:43:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:43:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 137 = 2023-12-05T22:43:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:43:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.status 2023-12-05T22:43:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.status 2023-12-05T22:43:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-05T22:43:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=137 2023-12-05T22:43:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 137 = 137 2023-12-05T22:43:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-05T22:43:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-05T22:43:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_10': 2023-12-05T22:43:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:43:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129d4bb029a9 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129d4bb029a9 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:01 2023 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:01 2023 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:01 2023 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:43:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 0042551e-e667-4458-be32-9094ff2c9220 2023-12-05T22:43:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:43:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-05T22:43:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:43:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:43:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.export 2023-12-05T22:43:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.export 2023-12-05T22:43:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.export 2023-12-05T22:43:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.export 2023-12-05T22:43:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.export 2023-12-05T22:43:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.export 2023-12-05T22:43:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_10.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_10.export 2023-12-05T22:43:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2023-12-05T22:43:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-05T22:43:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:43:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-05T22:43:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-05T22:43:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:43:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:43:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:43:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirror statusgrep mirror/image_11 -i 2023-12-05T22:43:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:43:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:43:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:43:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:43:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:43:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-05T22:43:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-05T22:43:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:43:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:43:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:43:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-05T22:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_11.status 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_11.status 2023-12-05T22:43:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.status 2023-12-05T22:43:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_11.status 2023-12-05T22:43:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_11.status 2023-12-05T22:43:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:43:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.status 2023-12-05T22:43:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-05T22:43:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-05T22:43:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 139 2023-12-05T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.status 2023-12-05T22:43:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.status 2023-12-05T22:43:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-05T22:43:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:43:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=139 2023-12-05T22:43:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:43:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.status 2023-12-05T22:43:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.status 2023-12-05T22:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-05T22:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=78 2023-12-05T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 139 = 78 2023-12-05T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:43:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.status 2023-12-05T22:43:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.status 2023-12-05T22:43:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-05T22:43:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:43:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:43:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:43:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:43:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:43:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 139 = 2023-12-05T22:43:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:43:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.status 2023-12-05T22:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.status 2023-12-05T22:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-05T22:43:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:43:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=139 2023-12-05T22:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 139 = 139 2023-12-05T22:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-05T22:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-05T22:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_11': 2023-12-05T22:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114014ca2f44 2023-12-05T22:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114014ca2f44 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:13 2023 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:13 2023 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:13 2023 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 08d57849-e22c-4949-ad8e-26b3cce133fe 2023-12-05T22:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:43:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-05T22:43:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:43:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:43:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.export 2023-12-05T22:43:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.export 2023-12-05T22:43:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.export 2023-12-05T22:43:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.export 2023-12-05T22:43:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.export 2023-12-05T22:43:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:43:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.export 2023-12-05T22:43:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_11.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_11.export 2023-12-05T22:43:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:43:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2023-12-05T22:43:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-05T22:43:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:43:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:43:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:43:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:43:35.370 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3595944670514686) > weight (0.25) 2023-12-05T22:43:35.370 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-05T22:43:35.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-05T22:43:35.371 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-05T22:43:35.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:43:35.374+0000 7fd70088b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 43017) UID: 1000 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:43:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:43:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-05T22:43:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:43:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:43:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:43:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:43:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_12grep 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: -i state 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:43:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:43:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-05T22:43:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-05T22:43:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-05T22:43:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:43:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-05T22:43:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-05T22:43:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-05T22:43:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:43:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:43:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:43:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-05T22:43:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-05T22:43:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:43:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:43:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-05T22:43:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:43:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-05T22:43:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:43:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-05T22:43:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-05T22:43:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-05T22:43:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:43:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:43:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:43:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-05T22:43:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:43:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-05T22:43:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:43:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-05T22:43:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:43:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-05T22:43:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:43:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-05T22:43:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-05T22:43:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:43:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:43:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:43:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:43:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:43:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-05T22:43:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:43:44.372 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-05T22:43:44.372 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:43:44.372 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-05T22:43:44.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-05T22:43:44.373 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-05T22:43:44.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-05T22:43:44.376 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-12-05T22:43:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-05T22:43:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-05T22:43:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-05T22:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:43:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-05T22:43:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-05T22:43:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:43:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:43:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:43:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:43:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:43:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:43:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-05T22:43:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:43:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-05T22:43:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:43:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:43:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:43:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-05T22:43:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:43:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:43:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:43:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:43:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-05T22:43:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:43:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:43:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:43:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:43:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:43:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:43:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:43:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-05T22:43:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:43:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:43:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:43:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:43:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-05T22:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-05T22:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:43:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:43:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:43:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-05T22:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:43:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:43:58.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8206886923037683) > weight (0.25) 2023-12-05T22:43:58.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8293157571591823) > weight (0.25) 2023-12-05T22:43:58.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3492273220009342) > weight (0.25) 2023-12-05T22:43:58.570 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-05T22:43:58.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-05T22:43:58.571 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-05T22:43:58.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:43:58.573+0000 7fdd3c021640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 40336) UID: 1000 2023-12-05T22:43:59.571 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-05T22:43:59.571 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:43:59.572 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-05T22:43:59.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-05T22:43:59.572 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-05T22:43:59.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-05T22:43:59.575 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-05T22:44:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-05T22:44:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-05T22:44:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-05T22:44:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:44:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:44:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-05T22:44:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:44:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:44:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-05T22:44:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_12.status 2023-12-05T22:44:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_12.status 2023-12-05T22:44:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.status 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_12.status 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_12.status 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.status 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-05T22:44:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 141 2023-12-05T22:44:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.status 2023-12-05T22:44:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.status 2023-12-05T22:44:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-05T22:44:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=141 2023-12-05T22:44:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.status 2023-12-05T22:44:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.status 2023-12-05T22:44:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-05T22:44:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 141 = 2023-12-05T22:44:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.status 2023-12-05T22:44:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.status 2023-12-05T22:44:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-05T22:44:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 141 = 2023-12-05T22:44:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.status 2023-12-05T22:44:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.status 2023-12-05T22:44:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-05T22:44:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=141 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 141 = 141 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-05T22:44:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_12': 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129deea11458 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129deea11458 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:21 2023 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:21 2023 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:21 2023 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:44:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:44:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 73016fb1-14a6-445e-a3b2-f2065161bec3 2023-12-05T22:44:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.export 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.export 2023-12-05T22:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.export 2023-12-05T22:44:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.export 2023-12-05T22:44:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.export 2023-12-05T22:44:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.export 2023-12-05T22:44:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_12.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_12.export 2023-12-05T22:44:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2023-12-05T22:44:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-05T22:44:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:44:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-05T22:44:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-05T22:44:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ localgrep loop_instance -i 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:44:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-05T22:44:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:44:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:44:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_13.status 2023-12-05T22:44:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_13.status 2023-12-05T22:44:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.status 2023-12-05T22:44:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_13.status 2023-12-05T22:44:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_13.status 2023-12-05T22:44:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.status 2023-12-05T22:44:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-05T22:44:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-05T22:44:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 143 2023-12-05T22:44:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.status 2023-12-05T22:44:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.status 2023-12-05T22:44:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-05T22:44:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=143 2023-12-05T22:44:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.status 2023-12-05T22:44:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.status 2023-12-05T22:44:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-05T22:44:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=80 2023-12-05T22:44:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 143 = 80 2023-12-05T22:44:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.status 2023-12-05T22:44:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.status 2023-12-05T22:44:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-05T22:44:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 143 = 2023-12-05T22:44:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.status 2023-12-05T22:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.status 2023-12-05T22:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-05T22:44:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=143 2023-12-05T22:44:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 143 = 143 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-05T22:44:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_13': 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1140ddf83b3a 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1140ddf83b3a 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:36 2023 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:36 2023 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:36 2023 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:44:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:44:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:44:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d5db6364-6a7e-48b6-a251-4e792c42f0d9 2023-12-05T22:44:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:44:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.export 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.export 2023-12-05T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.export 2023-12-05T22:44:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.export 2023-12-05T22:44:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.export 2023-12-05T22:44:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.export 2023-12-05T22:44:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_13.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_13.export 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:44:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-05T22:44:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-05T22:44:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:44:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-05T22:44:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:44:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:44:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep+ -ilocal state cluster=cluster1 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-05T22:44:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-05T22:44:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:44:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:44:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:44:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-05T22:44:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-05T22:44:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:44:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:44:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:44:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:44:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:44:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:44:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:44:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-12-05T22:44:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:44:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:44:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_14.status 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_14.status 2023-12-05T22:44:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.status 2023-12-05T22:44:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_14.status 2023-12-05T22:44:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_14.status 2023-12-05T22:44:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-05T22:44:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-05T22:44:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 145 2023-12-05T22:44:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.status 2023-12-05T22:44:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.status 2023-12-05T22:44:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-05T22:44:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=145 2023-12-05T22:44:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-05T22:44:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=81 2023-12-05T22:44:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 145 = 81 2023-12-05T22:44:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-05T22:44:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 145 = 2023-12-05T22:44:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-05T22:44:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 145 = 2023-12-05T22:44:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:44:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.status 2023-12-05T22:44:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-05T22:44:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=145 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 145 = 145 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_14': 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1225e165788a 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1225e165788a 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:44:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:42 2023 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:42 2023 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:42 2023 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c9eb233a-812a-48e1-98fd-e7779e2fce61 2023-12-05T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.export 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.export 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.export 2023-12-05T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.export 2023-12-05T22:44:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.export 2023-12-05T22:44:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.export 2023-12-05T22:44:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_14.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_14.export 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:44:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-05T22:44:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ greplocal -i instance=cluster1 2023-12-05T22:44:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:44:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:44:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:44:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-05T22:44:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-05T22:44:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:44:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:44:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-05T22:44:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_15.status 2023-12-05T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_15.status 2023-12-05T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.status 2023-12-05T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_15.status 2023-12-05T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_15.status 2023-12-05T22:44:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.status 2023-12-05T22:44:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-05T22:44:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-05T22:44:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 147 2023-12-05T22:44:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.status 2023-12-05T22:44:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.status 2023-12-05T22:44:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-05T22:44:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=147 2023-12-05T22:44:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.status 2023-12-05T22:44:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.status 2023-12-05T22:44:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-05T22:44:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 147 = 2023-12-05T22:44:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.status 2023-12-05T22:44:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.status 2023-12-05T22:44:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-05T22:44:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 147 = 2023-12-05T22:44:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.status 2023-12-05T22:44:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.status 2023-12-05T22:44:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-05T22:44:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=147 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 147 = 147 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_15': 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289eebd63d9 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289eebd63d9 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:34:53 2023 2023-12-05T22:44:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:34:53 2023 2023-12-05T22:44:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:34:53 2023 2023-12-05T22:44:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:44:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:44:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:44:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b9b1e588-b890-4a39-a107-1d78081756ec 2023-12-05T22:44:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:44:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.export 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.export 2023-12-05T22:44:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.export 2023-12-05T22:44:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.export 2023-12-05T22:44:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.export 2023-12-05T22:44:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.export 2023-12-05T22:44:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_15.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_15.export 2023-12-05T22:44:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2023-12-05T22:44:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-05T22:44:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-05T22:44:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:44:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:44:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:44:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster=cluster1 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ greplocal -i loop_instance state 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-05T22:44:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-05T22:44:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-05T22:44:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:44:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:44:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-12-05T22:44:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-05T22:44:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:44:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:44:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2023-12-05T22:44:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:44:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:44:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:44:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:44:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:44:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:44:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:44:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2023-12-05T22:44:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:44:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:44:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_16.status 2023-12-05T22:44:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_16.status 2023-12-05T22:44:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.status 2023-12-05T22:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_16.status 2023-12-05T22:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_16.status 2023-12-05T22:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.status 2023-12-05T22:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-05T22:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-05T22:44:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 149 2023-12-05T22:44:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.status 2023-12-05T22:44:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.status 2023-12-05T22:44:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-05T22:44:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=149 2023-12-05T22:44:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.status 2023-12-05T22:44:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.status 2023-12-05T22:44:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-05T22:44:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 149 = 2023-12-05T22:44:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.status 2023-12-05T22:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.status 2023-12-05T22:44:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-05T22:44:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 149 = 2023-12-05T22:44:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.status 2023-12-05T22:44:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.status 2023-12-05T22:44:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-05T22:44:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=149 2023-12-05T22:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 149 = 149 2023-12-05T22:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-05T22:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_16': 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129dc6b48695 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129dc6b48695 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:44:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:00 2023 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:00 2023 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:00 2023 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b3ff1efc-2dee-4937-993c-fb819906b94f 2023-12-05T22:44:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.export 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.export 2023-12-05T22:44:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.export 2023-12-05T22:44:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.export 2023-12-05T22:44:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.export 2023-12-05T22:44:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.export 2023-12-05T22:44:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_16.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_16.export 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-05T22:44:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:44:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:44:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:44:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-05T22:44:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:44:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grepshift -i 2023-12-05T22:44:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:44:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:44:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:44:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:44:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-05T22:44:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-05T22:44:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:44:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:44:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-05T22:44:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_17.status 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_17.status 2023-12-05T22:44:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.status 2023-12-05T22:44:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_17.status 2023-12-05T22:44:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_17.status 2023-12-05T22:44:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.status 2023-12-05T22:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-05T22:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-05T22:44:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 151 2023-12-05T22:44:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.status 2023-12-05T22:44:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.status 2023-12-05T22:44:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-05T22:44:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=151 2023-12-05T22:44:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.status 2023-12-05T22:44:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.status 2023-12-05T22:44:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-05T22:44:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=84 2023-12-05T22:44:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 151 = 84 2023-12-05T22:44:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.status 2023-12-05T22:44:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.status 2023-12-05T22:44:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-05T22:44:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 151 = 2023-12-05T22:44:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:42.742 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:44:42.748+0000 7fe8efb13640 -1 reset not still connected to 0x55b616f14b60 2023-12-05T22:44:42.742 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:44:42.748+0000 7fe8efb13640 -1 reset not still connected to 0x55b616f152b0 2023-12-05T22:44:42.742 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:44:42.748+0000 7fe8efb13640 -1 reset not still connected to 0x55b61ecd8c30 2023-12-05T22:44:42.742 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:44:42.748+0000 7fe8efb13640 -1 reset not still connected to 0x55b62d852820 2023-12-05T22:44:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.status 2023-12-05T22:44:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.status 2023-12-05T22:44:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-05T22:44:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=151 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 151 = 151 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_17': 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12aed8e8827f 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12aed8e8827f 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:09 2023 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:09 2023 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:09 2023 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9f56dd88-b64c-4071-b253-84119be63f20 2023-12-05T22:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:44:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-05T22:44:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:44:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:44:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.export 2023-12-05T22:44:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.export 2023-12-05T22:44:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.export 2023-12-05T22:44:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.export 2023-12-05T22:44:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.export 2023-12-05T22:44:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.export 2023-12-05T22:44:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_17.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_17.export 2023-12-05T22:44:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2023-12-05T22:44:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-05T22:44:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:44:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-05T22:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:44:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-05T22:44:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-05T22:44:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:44:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:44:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-05T22:44:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:44:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:44:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:44:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-05T22:44:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grep -i Replaying 2023-12-05T22:44:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo "state": "Replaying" 2023-12-05T22:44:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_18.status 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_18.status 2023-12-05T22:44:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.status 2023-12-05T22:44:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_18.status 2023-12-05T22:44:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_18.status 2023-12-05T22:44:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-05T22:44:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-05T22:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 153 2023-12-05T22:44:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.status 2023-12-05T22:44:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.status 2023-12-05T22:44:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-05T22:44:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=153 2023-12-05T22:44:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-05T22:44:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=85 2023-12-05T22:44:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = 85 2023-12-05T22:44:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-05T22:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = 2023-12-05T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-05T22:44:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = 2023-12-05T22:44:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:44:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.status 2023-12-05T22:44:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-05T22:44:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=153 2023-12-05T22:44:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = 153 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_18': 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ae8bbf13f2 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ae8bbf13f2 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:44:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:22 2023 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:22 2023 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:22 2023 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: a058c447-ff5e-49d0-9009-af35f59afbb5 2023-12-05T22:44:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.export 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.export 2023-12-05T22:44:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.export 2023-12-05T22:44:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.export 2023-12-05T22:44:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.export 2023-12-05T22:44:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:44:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.export 2023-12-05T22:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_18.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_18.export 2023-12-05T22:44:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2023-12-05T22:44:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-05T22:44:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:44:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-05T22:44:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:44:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:44:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:44:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:44:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:44:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:44:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:44:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-05T22:44:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-05T22:44:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:44:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:44:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:44:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-05T22:44:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:44:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:44:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:44:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:44:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-05T22:44:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:44:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_19.status 2023-12-05T22:44:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_19.status 2023-12-05T22:44:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.status 2023-12-05T22:44:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_19.status 2023-12-05T22:44:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_19.status 2023-12-05T22:44:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:44:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.status 2023-12-05T22:44:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-05T22:44:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-05T22:44:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 155 2023-12-05T22:44:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.status 2023-12-05T22:44:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:44:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.status 2023-12-05T22:44:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-05T22:44:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:44:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=155 2023-12-05T22:44:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:44:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.status 2023-12-05T22:44:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.status 2023-12-05T22:44:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-05T22:44:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 155 = 2023-12-05T22:44:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:44:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.status 2023-12-05T22:44:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.status 2023-12-05T22:44:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-05T22:44:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:44:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:44:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:44:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:44:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 155 = 2023-12-05T22:44:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:44:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.status 2023-12-05T22:44:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.status 2023-12-05T22:44:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-05T22:44:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=155 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 155 = 155 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:44:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-05T22:45:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_19': 2023-12-05T22:45:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:45:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:45:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:45:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129de63d66bd 2023-12-05T22:45:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129de63d66bd 2023-12-05T22:45:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:29 2023 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:29 2023 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:29 2023 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e72c6cbf-3629-4ab9-855f-c2cdd06d81a7 2023-12-05T22:45:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.export 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.export 2023-12-05T22:45:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.export 2023-12-05T22:45:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.export 2023-12-05T22:45:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.export 2023-12-05T22:45:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:00.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.export 2023-12-05T22:45:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_19.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_19.export 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:45:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-05T22:45:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:45:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:45:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:45:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:45:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-05T22:45:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-05T22:45:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-05T22:45:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-05T22:45:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-05T22:45:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-05T22:45:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-05T22:45:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-05T22:45:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-05T22:45:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:45:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:45:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:45:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-05T22:45:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:45:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:45:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:45:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:45:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_20.status 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_20.status 2023-12-05T22:45:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.status 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_20.status 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_20.status 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-05T22:45:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 157 2023-12-05T22:45:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.status 2023-12-05T22:45:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.status 2023-12-05T22:45:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-05T22:45:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=157 2023-12-05T22:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:45:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-05T22:45:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=87 2023-12-05T22:45:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = 87 2023-12-05T22:45:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:45:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-05T22:45:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = 2023-12-05T22:45:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:45:06.120 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:45:06.120+0000 7f1618813640 -1 reset not still connected to 0x558e24727110 2023-12-05T22:45:06.120 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:45:06.120+0000 7f1618813640 -1 reset not still connected to 0x558e2566a5b0 2023-12-05T22:45:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-05T22:45:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = 2023-12-05T22:45:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:45:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.status 2023-12-05T22:45:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-05T22:45:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=157 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = 157 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-05T22:45:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_20': 2023-12-05T22:45:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:45:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:45:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:45:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ae240d70ff 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ae240d70ff 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:43 2023 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:43 2023 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:43 2023 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1533eb9c-c4ac-468b-9060-8f72f4f33772 2023-12-05T22:45:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.export 2023-12-05T22:45:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.export 2023-12-05T22:45:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.export 2023-12-05T22:45:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.export 2023-12-05T22:45:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.export 2023-12-05T22:45:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.export 2023-12-05T22:45:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_20.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_20.export 2023-12-05T22:45:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2023-12-05T22:45:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-05T22:45:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:45:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift+ 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-05T22:45:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-05T22:45:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-05T22:45:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:12.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4934752144708039) > weight (0.25) 2023-12-05T22:45:12.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.47528351437621186) > weight (0.25) 2023-12-05T22:45:12.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8429590032870139) > weight (0.25) 2023-12-05T22:45:12.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.38220576537615103) > weight (0.25) 2023-12-05T22:45:12.576 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-12-05T22:45:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-12-05T22:45:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-05T22:45:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-12-05T22:45:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:45:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:45:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:45:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:45:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_21.status 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_21.status 2023-12-05T22:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.status 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_21.status 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_21.status 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.status 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-05T22:45:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 159 2023-12-05T22:45:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.status 2023-12-05T22:45:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.status 2023-12-05T22:45:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-05T22:45:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:45:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=159 2023-12-05T22:45:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:45:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.status 2023-12-05T22:45:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.status 2023-12-05T22:45:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-05T22:45:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 159 = 2023-12-05T22:45:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:45:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.status 2023-12-05T22:45:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.status 2023-12-05T22:45:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-05T22:45:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 159 = 2023-12-05T22:45:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:45:14.803 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:45:14.796+0000 7f4b0ce40640 -1 reset not still connected to 0x55b1153455f0 2023-12-05T22:45:14.804 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:45:14.796+0000 7f4b0ce40640 -1 reset not still connected to 0x55b11634dd40 2023-12-05T22:45:14.804 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:45:14.796+0000 7f4b0ce40640 -1 reset not still connected to 0x55b11a432a90 2023-12-05T22:45:14.804 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:45:14.796+0000 7f4b0ce40640 -1 reset not still connected to 0x55b11b5c1ad0 2023-12-05T22:45:14.804 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:45:14.796+0000 7f4b0ce40640 -1 reset not still connected to 0x55b11b5c41a0 2023-12-05T22:45:14.804 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:45:14.796+0000 7f4b0ce40640 -1 reset not still connected to 0x55b11b5c4270 2023-12-05T22:45:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.status 2023-12-05T22:45:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.status 2023-12-05T22:45:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-05T22:45:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=159 2023-12-05T22:45:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 159 = 159 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_21': 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1140f83b80c4 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1140f83b80c4 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:35:54 2023 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:35:54 2023 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:35:54 2023 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 18f502e6-d626-414b-b187-fc0882719611 2023-12-05T22:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.export 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.export 2023-12-05T22:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.export 2023-12-05T22:45:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.export 2023-12-05T22:45:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.export 2023-12-05T22:45:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.export 2023-12-05T22:45:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_21.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_21.export 2023-12-05T22:45:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2023-12-05T22:45:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-05T22:45:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:45:17.377 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-05T22:45:17.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-05T22:45:17.378 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-05T22:45:17.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:45:17.383+0000 7f9b29cbe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 52904) UID: 1000 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:45:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:19.379 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-05T22:45:19.379 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:45:19.379 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-05T22:45:19.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-05T22:45:19.380 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-05T22:45:19.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-05T22:45:19.383 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-05T22:45:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-05T22:45:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:45:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:45:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:45:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-05T22:45:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:45:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:45:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:45:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:45:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:45:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-05T22:45:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-05T22:45:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-05T22:45:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-05T22:45:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-05T22:45:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2023-12-05T22:45:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2023-12-05T22:45:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:45:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grep -i Replaying 2023-12-05T22:45:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo "state": "Replaying" 2023-12-05T22:45:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-05T22:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_22.status 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_22.status 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.status 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_22.status 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_22.status 2023-12-05T22:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-05T22:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-05T22:45:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 161 2023-12-05T22:45:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.status 2023-12-05T22:45:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.status 2023-12-05T22:45:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-05T22:45:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:45:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=161 2023-12-05T22:45:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:45:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-05T22:45:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=89 2023-12-05T22:45:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = 89 2023-12-05T22:45:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:45:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-05T22:45:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = 2023-12-05T22:45:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:45:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-05T22:45:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = 2023-12-05T22:45:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:45:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.status 2023-12-05T22:45:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-05T22:45:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=161 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = 161 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-05T22:45:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_22': 2023-12-05T22:45:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:45:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:45:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:45:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289a3eb145f 2023-12-05T22:45:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289a3eb145f 2023-12-05T22:45:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:45:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:45:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:45:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:45:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:07 2023 2023-12-05T22:45:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:07 2023 2023-12-05T22:45:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:07 2023 2023-12-05T22:45:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:45:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:45:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:45:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e296de3e-e023-46cc-9b13-df738af23268 2023-12-05T22:45:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.export 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.export 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.export 2023-12-05T22:45:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.export 2023-12-05T22:45:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.export 2023-12-05T22:45:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.export 2023-12-05T22:45:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_22.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_22.export 2023-12-05T22:45:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2023-12-05T22:45:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-05T22:45:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:45:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-05T22:45:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-05T22:45:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:45:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:45:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-05T22:45:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-05T22:45:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-05T22:45:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-05T22:45:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-05T22:45:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-12-05T22:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-12-05T22:45:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-05T22:45:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:45:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:45:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2023-12-05T22:45:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo "state": "Replaying" 2023-12-05T22:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i Replaying 2023-12-05T22:45:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-05T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_23.status 2023-12-05T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_23.status 2023-12-05T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.status 2023-12-05T22:45:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_23.status 2023-12-05T22:45:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_23.status 2023-12-05T22:45:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-05T22:45:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-05T22:45:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 163 2023-12-05T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.status 2023-12-05T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.status 2023-12-05T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-05T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:45:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=163 2023-12-05T22:45:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:45:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-05T22:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=90 2023-12-05T22:45:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 163 = 90 2023-12-05T22:45:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:45:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-05T22:45:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 163 = 2023-12-05T22:45:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-05T22:45:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 163 = 2023-12-05T22:45:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:45:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.status 2023-12-05T22:45:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-05T22:45:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=163 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 163 = 163 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_23': 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129d96efe6a8 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129d96efe6a8 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:45:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:13 2023 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:13 2023 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:13 2023 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e9b3d2ae-1928-4b14-803b-779daf73fd91 2023-12-05T22:45:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:45:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.export 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.export 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.export 2023-12-05T22:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.export 2023-12-05T22:45:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.export 2023-12-05T22:45:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:45:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.export 2023-12-05T22:45:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_23.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_23.export 2023-12-05T22:45:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2023-12-05T22:45:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-05T22:45:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:45:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-05T22:45:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-05T22:45:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:45:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-05T22:45:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:45:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:45:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd+ mirror status mirror/image_24 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep+ -ishift state 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-05T22:45:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-05T22:45:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-05T22:45:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-05T22:45:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:45:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-05T22:45:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-05T22:45:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-05T22:45:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-05T22:45:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:45:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-05T22:45:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:45:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-05T22:45:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-05T22:45:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-05T22:45:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-05T22:45:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:45:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-05T22:45:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:45:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-05T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-05T22:45:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:45:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:45:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:45:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:45:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-05T22:45:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-05T22:45:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:45:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:45:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:45:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:45:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-05T22:45:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:45:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:45:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:45:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:45:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-05T22:45:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-05T22:45:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:45:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:45:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:45:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:45:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:45:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:45:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:45:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:45:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-05T22:45:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:45:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:45:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:45:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:45:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-05T22:45:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:45:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_24.status 2023-12-05T22:45:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_24.status 2023-12-05T22:45:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.status 2023-12-05T22:45:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_24.status 2023-12-05T22:45:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_24.status 2023-12-05T22:45:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:45:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-05T22:45:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-05T22:45:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 165 2023-12-05T22:45:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.status 2023-12-05T22:45:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:45:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.status 2023-12-05T22:45:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-05T22:45:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:45:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=165 2023-12-05T22:45:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:45:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-05T22:45:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=91 2023-12-05T22:45:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 165 = 91 2023-12-05T22:45:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:45:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-05T22:45:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 165 = 2023-12-05T22:45:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:45:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-05T22:45:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:57.152 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:45:57.151+0000 7efd5eea3640 -1 reset not still connected to 0x55aea62f2dd0 2023-12-05T22:45:57.152 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:45:57.151+0000 7efd5eea3640 -1 reset not still connected to 0x55aea7621ba0 2023-12-05T22:45:57.152 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:45:57.151+0000 7efd5eea3640 -1 reset not still connected to 0x55aea7bcdad0 2023-12-05T22:45:57.152 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:45:57.151+0000 7efd5eea3640 -1 reset not still connected to 0x55aea8ffcd00 2023-12-05T22:45:57.152 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:45:57.151+0000 7efd5eea3640 -1 reset not still connected to 0x55aea9ed6680 2023-12-05T22:45:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:45:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:45:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:45:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:45:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:45:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 165 = 2023-12-05T22:45:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:45:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.status 2023-12-05T22:45:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-05T22:45:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=165 2023-12-05T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 165 = 165 2023-12-05T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-05T22:45:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:45:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:45:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:45:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_24': 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12aea469b8b1 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12aea469b8b1 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:45:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:19 2023 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:19 2023 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:19 2023 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 17202bda-24c3-40df-b59d-9904509b75df 2023-12-05T22:45:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:45:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:45:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.export 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.export 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.export 2023-12-05T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.export 2023-12-05T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.export 2023-12-05T22:46:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.export 2023-12-05T22:46:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_24.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_24.export 2023-12-05T22:46:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2023-12-05T22:46:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-05T22:46:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-05T22:46:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-05T22:46:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-05T22:46:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-05T22:46:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:46:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:46:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep+ -ilocal state loop_instance 2023-12-05T22:46:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-05T22:46:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-05T22:46:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2023-12-05T22:46:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:46:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:46:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2023-12-05T22:46:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:46:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:46:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_25.status 2023-12-05T22:46:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_25.status 2023-12-05T22:46:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.status 2023-12-05T22:46:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_25.status 2023-12-05T22:46:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_25.status 2023-12-05T22:46:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.status 2023-12-05T22:46:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-05T22:46:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-05T22:46:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 167 2023-12-05T22:46:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.status 2023-12-05T22:46:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.status 2023-12-05T22:46:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-05T22:46:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=167 2023-12-05T22:46:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.status 2023-12-05T22:46:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.status 2023-12-05T22:46:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-05T22:46:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 167 = 2023-12-05T22:46:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.status 2023-12-05T22:46:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.status 2023-12-05T22:46:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-05T22:46:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 167 = 2023-12-05T22:46:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.status 2023-12-05T22:46:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.status 2023-12-05T22:46:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-05T22:46:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=167 2023-12-05T22:46:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 167 = 167 2023-12-05T22:46:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-05T22:46:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_25': 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 11408786e694 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.11408786e694 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:46:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:29 2023 2023-12-05T22:46:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:29 2023 2023-12-05T22:46:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:29 2023 2023-12-05T22:46:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:46:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:46:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:46:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4ff38d91-f124-4824-8d8b-0b2a3eb228a9 2023-12-05T22:46:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:46:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-05T22:46:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:46:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:46:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.export 2023-12-05T22:46:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.export 2023-12-05T22:46:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.export 2023-12-05T22:46:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.export 2023-12-05T22:46:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.export 2023-12-05T22:46:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.export 2023-12-05T22:46:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_25.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_25.export 2023-12-05T22:46:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2023-12-05T22:46:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-05T22:46:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-05T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_26+ 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-05T22:46:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-05T22:46:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-05T22:46:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-05T22:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-05T22:46:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-05T22:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:46:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2023-12-05T22:46:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying"+ 2023-12-05T22:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i Replaying 2023-12-05T22:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_26.status 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_26.status 2023-12-05T22:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.status 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_26.status 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_26.status 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.status 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-05T22:46:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 169 2023-12-05T22:46:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.status 2023-12-05T22:46:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.status 2023-12-05T22:46:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-05T22:46:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=169 2023-12-05T22:46:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.status 2023-12-05T22:46:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.status 2023-12-05T22:46:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-05T22:46:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=93 2023-12-05T22:46:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 169 = 93 2023-12-05T22:46:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.status 2023-12-05T22:46:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.status 2023-12-05T22:46:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-05T22:46:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 169 = 2023-12-05T22:46:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.status 2023-12-05T22:46:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.status 2023-12-05T22:46:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-05T22:46:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=169 2023-12-05T22:46:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 169 = 169 2023-12-05T22:46:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-05T22:46:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_26': 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289c69d1a37 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289c69d1a37 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:46:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:39 2023 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:39 2023 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:39 2023 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 119a68f7-e780-449a-acf3-897b2c081275 2023-12-05T22:46:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:46:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-05T22:46:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:46:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:46:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.export 2023-12-05T22:46:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.export 2023-12-05T22:46:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.export 2023-12-05T22:46:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.export 2023-12-05T22:46:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.export 2023-12-05T22:46:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.export 2023-12-05T22:46:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_26.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_26.export 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:16.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-05T22:46:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-05T22:46:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:46:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:46:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:46:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-05T22:46:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-05T22:46:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-12-05T22:46:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:46:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:46:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-05T22:46:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_27.status 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_27.status 2023-12-05T22:46:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.status 2023-12-05T22:46:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_27.status 2023-12-05T22:46:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo cluster1-mirror-image_27.status 2023-12-05T22:46:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -e s|[/ ]|_|g 2023-12-05T22:46:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-05T22:46:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-05T22:46:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 171 2023-12-05T22:46:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.status 2023-12-05T22:46:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.status 2023-12-05T22:46:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-05T22:46:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=171 2023-12-05T22:46:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-05T22:46:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=94 2023-12-05T22:46:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = 94 2023-12-05T22:46:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-05T22:46:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = 2023-12-05T22:46:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-05T22:46:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = 2023-12-05T22:46:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:46:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.status 2023-12-05T22:46:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-05T22:46:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=171 2023-12-05T22:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = 171 2023-12-05T22:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-05T22:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-05T22:46:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_27': 2023-12-05T22:46:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:46:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:46:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:46:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129d3247bca9 2023-12-05T22:46:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129d3247bca9 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:36:53 2023 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:36:53 2023 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:36:53 2023 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d5a19c81-aa40-411c-8f9d-e68f5717dee7 2023-12-05T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.export 2023-12-05T22:46:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.export 2023-12-05T22:46:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.export 2023-12-05T22:46:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.export 2023-12-05T22:46:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.export 2023-12-05T22:46:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.export 2023-12-05T22:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_27.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_27.export 2023-12-05T22:46:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2023-12-05T22:46:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-05T22:46:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-05T22:46:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons+ cluster1 rbd mirror status mirror/image_28 2023-12-05T22:46:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i+ state 2023-12-05T22:46:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:local cluster_instance=cluster1 2023-12-05T22:46:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:46:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-05T22:46:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-05T22:46:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-05T22:46:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-12-05T22:46:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:46:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:46:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-05T22:46:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_28.status 2023-12-05T22:46:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_28.status 2023-12-05T22:46:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.status 2023-12-05T22:46:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_28.status 2023-12-05T22:46:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_28.status 2023-12-05T22:46:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.status 2023-12-05T22:46:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-05T22:46:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-05T22:46:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 173 2023-12-05T22:46:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.status 2023-12-05T22:46:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.status 2023-12-05T22:46:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-05T22:46:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=173 2023-12-05T22:46:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.status 2023-12-05T22:46:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.status 2023-12-05T22:46:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-05T22:46:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 173 = 2023-12-05T22:46:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.status 2023-12-05T22:46:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.status 2023-12-05T22:46:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-05T22:46:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 173 = 2023-12-05T22:46:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.status 2023-12-05T22:46:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.status 2023-12-05T22:46:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-05T22:46:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=173 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 173 = 173 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-05T22:46:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_28': 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12aeb7173633 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12aeb7173633 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:07 2023 2023-12-05T22:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:07 2023 2023-12-05T22:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:07 2023 2023-12-05T22:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 188e2fd3-baca-493b-8be5-724525683def 2023-12-05T22:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.export 2023-12-05T22:46:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.export 2023-12-05T22:46:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.export 2023-12-05T22:46:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.export 2023-12-05T22:46:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.export 2023-12-05T22:46:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.export 2023-12-05T22:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_28.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_28.export 2023-12-05T22:46:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2023-12-05T22:46:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-05T22:46:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-05T22:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-05T22:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-05T22:46:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-05T22:46:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-05T22:46:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-05T22:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-05T22:46:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:46:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:46:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_29.status 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_29.status 2023-12-05T22:46:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.status 2023-12-05T22:46:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_29.status 2023-12-05T22:46:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_29.status 2023-12-05T22:46:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.status 2023-12-05T22:46:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-05T22:46:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-05T22:46:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 175 2023-12-05T22:46:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.status 2023-12-05T22:46:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.status 2023-12-05T22:46:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-05T22:46:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=175 2023-12-05T22:46:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.status 2023-12-05T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.status 2023-12-05T22:46:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-05T22:46:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 175 = 2023-12-05T22:46:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.status 2023-12-05T22:46:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.status 2023-12-05T22:46:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-05T22:46:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 175 = 2023-12-05T22:46:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.status 2023-12-05T22:46:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.status 2023-12-05T22:46:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-05T22:46:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=175 2023-12-05T22:46:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 175 = 175 2023-12-05T22:46:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-05T22:46:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-05T22:46:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_29': 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 11406dc35f72 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.11406dc35f72 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:16 2023 2023-12-05T22:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:16 2023 2023-12-05T22:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:16 2023 2023-12-05T22:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4f42a30e-47bf-4a43-ad22-e89c1569217d 2023-12-05T22:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-05T22:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.export 2023-12-05T22:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.export 2023-12-05T22:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.export 2023-12-05T22:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.export 2023-12-05T22:46:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.export 2023-12-05T22:46:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.export 2023-12-05T22:46:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_29.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_29.export 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-05T22:46:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-05T22:46:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-05T22:46:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1+ 2023-12-05T22:46:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:46:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:46:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-05T22:46:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-05T22:46:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-05T22:46:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-05T22:46:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-05T22:46:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:46:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:46:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_30.status 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_30.status 2023-12-05T22:46:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.status 2023-12-05T22:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_30.status 2023-12-05T22:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_30.status 2023-12-05T22:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.status 2023-12-05T22:46:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-05T22:46:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-05T22:46:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 177 2023-12-05T22:46:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.status 2023-12-05T22:46:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.status 2023-12-05T22:46:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-05T22:46:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=177 2023-12-05T22:46:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.status 2023-12-05T22:46:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.status 2023-12-05T22:46:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-05T22:46:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=97 2023-12-05T22:46:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 177 = 97 2023-12-05T22:46:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.status 2023-12-05T22:46:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.status 2023-12-05T22:46:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-05T22:46:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 177 = 2023-12-05T22:46:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.status 2023-12-05T22:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.status 2023-12-05T22:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-05T22:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=177 2023-12-05T22:46:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 177 = 177 2023-12-05T22:46:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-05T22:46:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_30': 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289bb990f87 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289bb990f87 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:46:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:23 2023 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:23 2023 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:23 2023 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 607abc1b-0895-4b2d-abb8-f771bc5c407f 2023-12-05T22:46:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:46:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-05T22:46:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:46:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:46:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.export 2023-12-05T22:46:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.export 2023-12-05T22:46:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.export 2023-12-05T22:46:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.export 2023-12-05T22:46:44.384 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.707434723409376) > weight (0.25) 2023-12-05T22:46:44.384 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-05T22:46:44.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-05T22:46:44.385 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-05T22:46:44.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:46:44.390+0000 7f8fba477640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 56555) UID: 1000 2023-12-05T22:46:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.export 2023-12-05T22:46:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.export 2023-12-05T22:46:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_30.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_30.export 2023-12-05T22:46:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2023-12-05T22:46:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-05T22:46:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-05T22:46:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-05T22:46:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-05T22:46:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-12-05T22:46:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:46:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2023-12-05T22:46:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:46:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:46:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_31.status 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_31.status 2023-12-05T22:46:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.status 2023-12-05T22:46:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_31.status 2023-12-05T22:46:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_31.status 2023-12-05T22:46:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-05T22:46:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-05T22:46:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 179 2023-12-05T22:46:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.status 2023-12-05T22:46:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.status 2023-12-05T22:46:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-05T22:46:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=179 2023-12-05T22:46:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-05T22:46:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=98 2023-12-05T22:46:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 179 = 98 2023-12-05T22:46:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-05T22:46:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 179 = 2023-12-05T22:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-05T22:46:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:46:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:46:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:46:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:46:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 179 = 2023-12-05T22:46:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:46:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.status 2023-12-05T22:46:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-05T22:46:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=179 2023-12-05T22:46:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 179 = 179 2023-12-05T22:46:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-05T22:46:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:52.386 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-05T22:46:52.386 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:46:52.386 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-05T22:46:52.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-05T22:46:52.387 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-05T22:46:52.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-05T22:46:52.390 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-05T22:46:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_31': 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129ddaa1bf51 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129ddaa1bf51 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:46:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:31 2023 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:31 2023 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:31 2023 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 36786ed1-b139-4c21-bcaa-0cacd1cad4dc 2023-12-05T22:46:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.export 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.export 2023-12-05T22:46:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.export 2023-12-05T22:46:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.export 2023-12-05T22:46:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.export 2023-12-05T22:46:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:46:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.export 2023-12-05T22:46:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_31.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_31.export 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:46:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:46:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-05T22:46:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-05T22:46:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:46:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:46:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:46:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-05T22:46:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:46:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:46:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:46:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:46:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:46:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:46:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:46:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-05T22:46:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:46:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:46:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:46:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:46:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:46:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-05T22:46:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-05T22:46:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:46:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:46:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:46:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:46:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:46:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-12-05T22:46:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-05T22:46:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:46:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:46:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:46:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:46:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:46:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-12-05T22:46:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-05T22:46:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:46:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:46:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:46:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:46:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:46:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:46:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:46:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:46:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:46:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:46:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:46:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-12-05T22:46:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:46:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:46:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:46:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_32.status 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_32.status 2023-12-05T22:46:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.status 2023-12-05T22:46:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_32.status 2023-12-05T22:46:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_32.status 2023-12-05T22:46:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:46:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:46:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-05T22:46:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-05T22:46:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 181 2023-12-05T22:46:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.status 2023-12-05T22:46:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:46:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.status 2023-12-05T22:46:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-05T22:46:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:46:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=181 2023-12-05T22:46:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:46:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:46:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:46:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:46:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-05T22:46:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=99 2023-12-05T22:46:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = 99 2023-12-05T22:46:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:46:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:46:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:46:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-05T22:46:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=99 2023-12-05T22:46:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = 99 2023-12-05T22:46:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:46:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:46:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:46:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:46:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:46:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:46:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-05T22:46:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:46:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:46:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=99 2023-12-05T22:46:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = 99 2023-12-05T22:46:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:47:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:47:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:47:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:47:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-05T22:47:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = 2023-12-05T22:47:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:47:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:47:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:47:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.status 2023-12-05T22:47:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-05T22:47:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=181 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = 181 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:03.577 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5140986522598414) > weight (0.25) 2023-12-05T22:47:03.577 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:47:03.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:47:03.578 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-05T22:47:03.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:47:03.585+0000 7fc3b130b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 49603) UID: 1000 2023-12-05T22:47:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_32': 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ae48651e2e 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ae48651e2e 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:44 2023 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:44 2023 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:44 2023 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 726117d0-c60f-4f67-b079-12119b0741a6 2023-12-05T22:47:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:47:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-05T22:47:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:47:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:47:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.export 2023-12-05T22:47:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.export 2023-12-05T22:47:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.export 2023-12-05T22:47:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.export 2023-12-05T22:47:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.export 2023-12-05T22:47:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.export 2023-12-05T22:47:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_32.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_32.export 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:47:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-05T22:47:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:47:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:47:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:47:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd+ mirror status mirror/image_33 2023-12-05T22:47:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ greplocal -i cluster_instance=cluster1 state 2023-12-05T22:47:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:47:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:47:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:47:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-05T22:47:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-05T22:47:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-05T22:47:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:47:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-12-05T22:47:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:47:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2023-12-05T22:47:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-05T22:47:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-05T22:47:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-05T22:47:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:47:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-12-05T22:47:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2023-12-05T22:47:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:47:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:47:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:47:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2023-12-05T22:47:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:47:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-05T22:47:11.578 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:47:11.578 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:47:11.579 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:47:11.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:47:11.579 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:47:11.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:47:11.581 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-05T22:47:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-05T22:47:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:47:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-12-05T22:47:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:47:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2023-12-05T22:47:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:47:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-05T22:47:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-05T22:47:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-05T22:47:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-05T22:47:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:47:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:47:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:47:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:47:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_33.status 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_33.status 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.status 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_33.status 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_33.status 2023-12-05T22:47:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-05T22:47:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-05T22:47:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 183 2023-12-05T22:47:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.status 2023-12-05T22:47:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.status 2023-12-05T22:47:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-05T22:47:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:47:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=183 2023-12-05T22:47:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:47:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-05T22:47:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=100 2023-12-05T22:47:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 183 = 100 2023-12-05T22:47:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:47:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-05T22:47:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 183 = 2023-12-05T22:47:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:47:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-05T22:47:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 183 = 2023-12-05T22:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:47:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.status 2023-12-05T22:47:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-05T22:47:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=183 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 183 = 183 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-05T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_33': 2023-12-05T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1140724b4272 2023-12-05T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1140724b4272 2023-12-05T22:47:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:37:50 2023 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:37:50 2023 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:37:50 2023 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 80cd4891-b460-4331-9eae-304b773fde31 2023-12-05T22:47:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:47:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-05T22:47:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:47:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:47:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.export 2023-12-05T22:47:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.export 2023-12-05T22:47:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.export 2023-12-05T22:47:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.export 2023-12-05T22:47:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.export 2023-12-05T22:47:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.export 2023-12-05T22:47:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_33.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_33.export 2023-12-05T22:47:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2023-12-05T22:47:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-05T22:47:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-05T22:47:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:47:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:47:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:47:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-05T22:47:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:47:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:47:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:47:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:47:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i state 2023-12-05T22:47:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-05T22:47:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-05T22:47:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-05T22:47:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-05T22:47:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:47:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grep -i Replaying 2023-12-05T22:47:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo "state": "Replaying" 2023-12-05T22:47:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_34.status 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_34.status 2023-12-05T22:47:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.status 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_34.status 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_34.status 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-05T22:47:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 185 2023-12-05T22:47:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.status 2023-12-05T22:47:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.status 2023-12-05T22:47:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-05T22:47:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:47:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=185 2023-12-05T22:47:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:47:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-05T22:47:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=101 2023-12-05T22:47:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = 101 2023-12-05T22:47:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:47:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-05T22:47:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = 2023-12-05T22:47:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:47:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-05T22:47:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = 2023-12-05T22:47:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:47:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.status 2023-12-05T22:47:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-05T22:47:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=185 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = 185 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-05T22:47:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_34': 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289ed78aa5 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289ed78aa5 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:03 2023 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:03 2023 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:03 2023 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:47:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:47:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:47:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c95cf52a-6e99-4b13-94b9-00efacfbc9bf 2023-12-05T22:47:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:47:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-05T22:47:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:47:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:47:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.export 2023-12-05T22:47:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.export 2023-12-05T22:47:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.export 2023-12-05T22:47:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.export 2023-12-05T22:47:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.export 2023-12-05T22:47:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.export 2023-12-05T22:47:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_34.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_34.export 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:47:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-05T22:47:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:47:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:47:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:47:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1+ rbd mirror status mirror/image_35 2023-12-05T22:47:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep+ -i statelocal 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: cluster_instance=cluster1 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-05T22:47:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-05T22:47:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-12-05T22:47:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-12-05T22:47:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-05T22:47:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2023-12-05T22:47:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:47:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:47:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:47:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:47:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:47:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:47:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2023-12-05T22:47:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:47:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:47:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:47:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-05T22:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_35.status 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_35.status 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.status 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_35.status 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_35.status 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-05T22:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-05T22:47:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 187 2023-12-05T22:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.status 2023-12-05T22:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.status 2023-12-05T22:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-05T22:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:47:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=187 2023-12-05T22:47:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:47:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-05T22:47:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=102 2023-12-05T22:47:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 187 = 102 2023-12-05T22:47:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:47:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-05T22:47:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 187 = 2023-12-05T22:47:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:47:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-05T22:47:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 187 = 2023-12-05T22:47:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:47:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.status 2023-12-05T22:47:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-05T22:47:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=187 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 187 = 187 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-05T22:47:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_35': 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12c2a4cefb19 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12c2a4cefb19 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:16 2023 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:16 2023 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:16 2023 2023-12-05T22:47:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:47:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:47:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:47:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1d6600a5-89f1-4e94-812c-f425e83f649a 2023-12-05T22:47:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:47:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-05T22:47:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:47:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:47:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.export 2023-12-05T22:47:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.export 2023-12-05T22:47:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.export 2023-12-05T22:47:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.export 2023-12-05T22:47:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.export 2023-12-05T22:47:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.export 2023-12-05T22:47:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_35.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_35.export 2023-12-05T22:47:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2023-12-05T22:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-05T22:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:47:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-05T22:47:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-05T22:47:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:47:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:47:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:47:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grepadmin_daemons -i cluster1 state 2023-12-05T22:47:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: rbd mirror status mirror/image_36 2023-12-05T22:47:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:47:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:47:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:47:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:47:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-05T22:47:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-05T22:47:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-12-05T22:47:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-05T22:47:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2023-12-05T22:47:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:47:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:47:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:47:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:47:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:47:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2023-12-05T22:47:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:47:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:47:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:47:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-05T22:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_36.status 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo cluster2-mirror-image_36.statussed -e 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: s|[/ ]|_|g 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.status 2023-12-05T22:47:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_36.status 2023-12-05T22:47:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_36.status 2023-12-05T22:47:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.status 2023-12-05T22:47:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-05T22:47:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-05T22:47:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 189 2023-12-05T22:47:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.status 2023-12-05T22:47:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.status 2023-12-05T22:47:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-05T22:47:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:47:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=189 2023-12-05T22:47:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:47:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.status 2023-12-05T22:47:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.status 2023-12-05T22:47:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-05T22:47:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 189 = 2023-12-05T22:47:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:47:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.status 2023-12-05T22:47:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.status 2023-12-05T22:47:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-05T22:47:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 189 = 2023-12-05T22:47:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:47:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.status 2023-12-05T22:47:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.status 2023-12-05T22:47:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-05T22:47:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=189 2023-12-05T22:47:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 189 = 189 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-05T22:47:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_36': 2023-12-05T22:47:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:47:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:47:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:47:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12c2cd12ecad 2023-12-05T22:47:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12c2cd12ecad 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:30 2023 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:30 2023 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:30 2023 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4f5aa172-1810-4fc1-84bc-bfbb3a129dcd 2023-12-05T22:47:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:47:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-05T22:47:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:47:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:47:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.export 2023-12-05T22:47:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.export 2023-12-05T22:47:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.export 2023-12-05T22:47:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.export 2023-12-05T22:47:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.export 2023-12-05T22:47:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.export 2023-12-05T22:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_36.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_36.export 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:47:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-05T22:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-05T22:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:47:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-05T22:47:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:47:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:47:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:47:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-05T22:47:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:47:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1+ 2023-12-05T22:47:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:47:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:47:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:47:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:47:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-05T22:47:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-05T22:47:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-12-05T22:47:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-12-05T22:47:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:47:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:47:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:47:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:47:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:47:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_37.status 2023-12-05T22:47:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_37.status 2023-12-05T22:47:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.status 2023-12-05T22:47:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_37.status 2023-12-05T22:47:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_37.status 2023-12-05T22:47:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.status 2023-12-05T22:47:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-05T22:47:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-05T22:47:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 191 2023-12-05T22:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.status 2023-12-05T22:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.status 2023-12-05T22:47:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-05T22:47:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:47:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=191 2023-12-05T22:47:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:47:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.status 2023-12-05T22:47:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.status 2023-12-05T22:47:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-05T22:47:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=104 2023-12-05T22:47:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 191 = 104 2023-12-05T22:47:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:47:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.status 2023-12-05T22:47:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.status 2023-12-05T22:47:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-05T22:47:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 191 = 2023-12-05T22:47:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:47:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.status 2023-12-05T22:47:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.status 2023-12-05T22:47:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-05T22:47:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=191 2023-12-05T22:47:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 191 = 191 2023-12-05T22:47:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-05T22:47:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-05T22:47:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_37': 2023-12-05T22:47:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:47:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114098f0772b 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114098f0772b 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:36 2023 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:36 2023 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:36 2023 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:47:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:47:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 21086603-d365-40b9-9a1b-a87abe17542f 2023-12-05T22:47:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.export 2023-12-05T22:47:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.export 2023-12-05T22:47:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.export 2023-12-05T22:47:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.export 2023-12-05T22:47:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.export 2023-12-05T22:47:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:47:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.export 2023-12-05T22:47:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_37.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_37.export 2023-12-05T22:47:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2023-12-05T22:47:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-05T22:47:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:47:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:47:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-05T22:47:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:47:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:47:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:47:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-05T22:47:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:47:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster=cluster1 2023-12-05T22:47:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:47:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:47:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:47:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:47:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:47:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:47:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-05T22:47:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:47:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:47:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:47:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:47:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:47:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:47:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:47:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-05T22:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-05T22:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:47:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:47:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:47:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-12-05T22:47:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-05T22:47:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-12-05T22:47:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:47:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:47:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:47:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:47:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-12-05T22:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-05T22:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:47:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:47:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2023-12-05T22:47:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:47:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:47:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:47:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:47:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2023-12-05T22:47:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:47:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:47:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:47:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:47:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_38.status 2023-12-05T22:47:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_38.status 2023-12-05T22:47:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.status 2023-12-05T22:47:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_38.status 2023-12-05T22:47:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_38.status 2023-12-05T22:47:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-05T22:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-05T22:47:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 193 2023-12-05T22:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.status 2023-12-05T22:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.status 2023-12-05T22:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-05T22:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:47:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=193 2023-12-05T22:47:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:47:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:47:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:47:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-05T22:47:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=105 2023-12-05T22:47:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 193 = 105 2023-12-05T22:47:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-05T22:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:47:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:47:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:47:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:47:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:47:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:47:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:47:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 193 = 2023-12-05T22:47:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:48:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:48:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:48:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:48:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-05T22:48:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 193 = 2023-12-05T22:48:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.status 2023-12-05T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-05T22:48:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=193 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 193 = 193 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-05T22:48:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_38': 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289b38a1859 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289b38a1859 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:38:48 2023 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:38:48 2023 2023-12-05T22:48:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:38:48 2023 2023-12-05T22:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ffdb2cf0-ec49-4f59-9404-9cf65a1b49e0 2023-12-05T22:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:48:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-05T22:48:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:48:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:48:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.export 2023-12-05T22:48:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.export 2023-12-05T22:48:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.export 2023-12-05T22:48:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.export 2023-12-05T22:48:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.export 2023-12-05T22:48:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.export 2023-12-05T22:48:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_38.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_38.export 2023-12-05T22:48:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2023-12-05T22:48:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-05T22:48:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:48:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-05T22:48:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:48:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:48:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:48:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-12-05T22:48:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:48:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + shift 2023-12-05T22:48:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1grep 2023-12-05T22:48:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: -i state 2023-12-05T22:48:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:48:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:48:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:48:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-05T22:48:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-05T22:48:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:48:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:48:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:48:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:48:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:48:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:48:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-05T22:48:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:48:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:48:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-05T22:48:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_39.status 2023-12-05T22:48:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_39.status 2023-12-05T22:48:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.status 2023-12-05T22:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_39.status 2023-12-05T22:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_39.status 2023-12-05T22:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.status 2023-12-05T22:48:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-05T22:48:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-05T22:48:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 195 2023-12-05T22:48:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.status 2023-12-05T22:48:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.status 2023-12-05T22:48:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-12-05T22:48:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:48:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=195 2023-12-05T22:48:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:48:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.status 2023-12-05T22:48:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.status 2023-12-05T22:48:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-05T22:48:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 195 = 2023-12-05T22:48:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:48:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.status 2023-12-05T22:48:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.status 2023-12-05T22:48:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-05T22:48:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 195 = 2023-12-05T22:48:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:48:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.status 2023-12-05T22:48:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.status 2023-12-05T22:48:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-05T22:48:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=195 2023-12-05T22:48:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 195 = 195 2023-12-05T22:48:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-12-05T22:48:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-12-05T22:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_39': 2023-12-05T22:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12aefb608ff2 2023-12-05T22:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12aefb608ff2 2023-12-05T22:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:01 2023 2023-12-05T22:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:01 2023 2023-12-05T22:48:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:01 2023 2023-12-05T22:48:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:48:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:48:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:48:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b11ea2a8-7502-44da-ba95-ff96ab7a913f 2023-12-05T22:48:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:48:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_39 snap 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.export 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.export 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.export 2023-12-05T22:48:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.export 2023-12-05T22:48:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.export 2023-12-05T22:48:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.export 2023-12-05T22:48:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_39.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_39.export 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_40 snap 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:48:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-12-05T22:48:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:48:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:48:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1+ rbd mirror status mirror/image_40 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:48:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:48:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-12-05T22:48:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-05T22:48:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:48:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-12-05T22:48:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-05T22:48:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:48:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-12-05T22:48:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-05T22:48:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2023-12-05T22:48:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:48:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:48:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:48:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2023-12-05T22:48:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:48:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:48:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:48:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:48:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:48:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:48:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-12-05T22:48:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_40.status 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_40.status 2023-12-05T22:48:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.status 2023-12-05T22:48:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_40.status 2023-12-05T22:48:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_40.status 2023-12-05T22:48:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.status 2023-12-05T22:48:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-12-05T22:48:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-12-05T22:48:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 197 2023-12-05T22:48:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.status 2023-12-05T22:48:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.status 2023-12-05T22:48:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-12-05T22:48:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:48:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=197 2023-12-05T22:48:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:48:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.status 2023-12-05T22:48:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.status 2023-12-05T22:48:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-05T22:48:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 197 = 2023-12-05T22:48:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:48:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.status 2023-12-05T22:48:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.status 2023-12-05T22:48:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-05T22:48:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 197 = 2023-12-05T22:48:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:48:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.status 2023-12-05T22:48:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.status 2023-12-05T22:48:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-05T22:48:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=197 2023-12-05T22:48:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 197 = 197 2023-12-05T22:48:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-12-05T22:48:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-12-05T22:48:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_40': 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12c255b1e12a 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12c255b1e12a 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:08 2023 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:08 2023 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:08 2023 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:48:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:48:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:48:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 8480990e-2ffd-4b7c-89de-bb66a57203df 2023-12-05T22:48:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:48:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_40 snap 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.export 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.export 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.export 2023-12-05T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.export 2023-12-05T22:48:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.export 2023-12-05T22:48:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.export 2023-12-05T22:48:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_40.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_40.export 2023-12-05T22:48:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2023-12-05T22:48:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_41 snap 2023-12-05T22:48:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-12-05T22:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-12-05T22:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-12-05T22:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status+ mirror/image_41 2023-12-05T22:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:48:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:48:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:48:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-12-05T22:48:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-12-05T22:48:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-05T22:48:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-12-05T22:48:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:48:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:48:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-12-05T22:48:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-05T22:48:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2023-12-05T22:48:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:48:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:48:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:48:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:48:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:48:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:48:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2023-12-05T22:48:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-05T22:48:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:48:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2023-12-05T22:48:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:48:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:48:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:48:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:48:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:48:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:48:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-12-05T22:48:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:48:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:48:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-12-05T22:48:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_41.status 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_41.status 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.status 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_41.status 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_41.status 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-12-05T22:48:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 199 2023-12-05T22:48:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.status 2023-12-05T22:48:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.status 2023-12-05T22:48:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-12-05T22:48:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:48:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=199 2023-12-05T22:48:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:48:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-05T22:48:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=108 2023-12-05T22:48:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = 108 2023-12-05T22:48:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-05T22:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = 2023-12-05T22:48:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:48:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: mirror/image_41 --format xml 2023-12-05T22:48:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = 2023-12-05T22:48:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:48:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.status 2023-12-05T22:48:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-05T22:48:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=199 2023-12-05T22:48:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = 199 2023-12-05T22:48:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-12-05T22:48:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-12-05T22:48:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_41': 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1140d0916408 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1140d0916408 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:21 2023 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:21 2023 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:21 2023 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f48ece28-d198-4bf2-878b-dd40720a4c29 2023-12-05T22:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:48:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_41 snap 2023-12-05T22:48:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:48:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:48:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.export 2023-12-05T22:48:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.export 2023-12-05T22:48:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.export 2023-12-05T22:48:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.export 2023-12-05T22:48:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.export 2023-12-05T22:48:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.export 2023-12-05T22:48:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_41.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_41.export 2023-12-05T22:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2023-12-05T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_42 snap 2023-12-05T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:48:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-12-05T22:48:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-12-05T22:48:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state+ 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:local instance=cluster1 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-12-05T22:48:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:48:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:48:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:48:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_42.status 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_42.status 2023-12-05T22:48:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.status 2023-12-05T22:48:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_42.status 2023-12-05T22:48:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_42.status 2023-12-05T22:48:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.status 2023-12-05T22:48:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-12-05T22:48:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-12-05T22:48:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 201 2023-12-05T22:48:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.status 2023-12-05T22:48:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.status 2023-12-05T22:48:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-12-05T22:48:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:48:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=201 2023-12-05T22:48:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:48:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.status 2023-12-05T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.status 2023-12-05T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-05T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=109 2023-12-05T22:48:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 201 = 109 2023-12-05T22:48:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:48:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.status 2023-12-05T22:48:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.status 2023-12-05T22:48:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-05T22:48:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 201 = 2023-12-05T22:48:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:48:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.status 2023-12-05T22:48:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.status 2023-12-05T22:48:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-05T22:48:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=201 2023-12-05T22:48:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 201 = 201 2023-12-05T22:48:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-12-05T22:48:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-12-05T22:48:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_42': 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128915e90f7c 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128915e90f7c 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:36 2023 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:36 2023 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:36 2023 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:48:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:48:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 50eb144e-0936-49e6-8545-fb513c5ca376 2023-12-05T22:48:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:48:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_42 snap 2023-12-05T22:48:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:48:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:48:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.export 2023-12-05T22:48:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.export 2023-12-05T22:48:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.export 2023-12-05T22:48:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.export 2023-12-05T22:48:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.export 2023-12-05T22:48:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.export 2023-12-05T22:48:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_42.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_42.export 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_43 snap 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:48:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-12-05T22:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:48:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_43 2023-12-05T22:48:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:48:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:48:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-12-05T22:48:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-05T22:48:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:48:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-12-05T22:48:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:48:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + grep -i Replaying 2023-12-05T22:48:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:echo "state": "Replaying" 2023-12-05T22:48:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:48:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:48:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:48:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-12-05T22:48:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_43.status 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_43.status 2023-12-05T22:48:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.status 2023-12-05T22:48:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_43.status 2023-12-05T22:48:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_43.status 2023-12-05T22:48:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:48:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-12-05T22:48:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-12-05T22:48:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 203 2023-12-05T22:48:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.status 2023-12-05T22:48:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.status 2023-12-05T22:48:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-12-05T22:48:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:48:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=203 2023-12-05T22:48:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:48:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-05T22:48:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=110 2023-12-05T22:48:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = 110 2023-12-05T22:48:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:48:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-05T22:48:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = 2023-12-05T22:48:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:48:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-05T22:48:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:48:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:48:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:48:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:48:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:48:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = 2023-12-05T22:48:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:48:41.582 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-05T22:48:41.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-05T22:48:41.583 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-05T22:48:41.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:48:41.589+0000 7fb49331b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 50672) UID: 1000 2023-12-05T22:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.status 2023-12-05T22:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-05T22:48:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:48:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:48:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=203 2023-12-05T22:48:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = 203 2023-12-05T22:48:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-12-05T22:48:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:42.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3699999789461247) > weight (0.25) 2023-12-05T22:48:42.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8659588556797094) > weight (0.25) 2023-12-05T22:48:42.391 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-05T22:48:42.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-05T22:48:42.391 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-05T22:48:42.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:48:42.397+0000 7f4baa662640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 54010) UID: 1000 2023-12-05T22:48:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-12-05T22:48:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_43': 2023-12-05T22:48:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:48:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:48:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:48:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12aedcb2e53b 2023-12-05T22:48:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12aedcb2e53b 2023-12-05T22:48:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:48:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:48:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:48:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:48:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:42 2023 2023-12-05T22:48:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:42 2023 2023-12-05T22:48:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:42 2023 2023-12-05T22:48:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:48:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:48:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:48:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: a0acf697-0cfe-4b8b-880e-ea4251b3ce8c 2023-12-05T22:48:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:48:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_43 snap 2023-12-05T22:48:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:48:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:48:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.export 2023-12-05T22:48:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.export 2023-12-05T22:48:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.export 2023-12-05T22:48:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.export 2023-12-05T22:48:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.export 2023-12-05T22:48:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:48:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.export 2023-12-05T22:48:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_43.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_43.export 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_44 snap 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:48:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:48:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:48:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:48:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:45.583 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-05T22:48:45.583 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:48:45.584 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-05T22:48:45.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-05T22:48:45.584 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-05T22:48:45.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-05T22:48:45.587 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-05T22:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-12-05T22:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:48:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster=cluster1 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-05T22:48:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-05T22:48:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:48:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:48:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:48:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:48:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-05T22:48:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:48:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-05T22:48:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-05T22:48:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2023-12-05T22:48:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:48:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:48:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:48:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2023-12-05T22:48:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:48:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-05T22:48:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-05T22:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:48:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-05T22:48:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-05T22:48:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:48:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-05T22:48:48.392 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-05T22:48:48.392 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:48:48.393 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-05T22:48:48.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-05T22:48:48.393 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-05T22:48:48.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-05T22:48:48.396 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-12-05T22:48:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:48:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2023-12-05T22:48:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:48:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-05T22:48:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:48:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-05T22:48:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:48:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-05T22:48:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:48:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2023-12-05T22:48:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:48:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:48:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:48:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:48:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:48:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:48:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-05T22:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:48:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:48:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:48:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:48:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:48:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-05T22:48:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-05T22:48:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2023-12-05T22:48:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:48:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:48:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:48:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:48:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:48:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:48:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:48:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:48:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2023-12-05T22:48:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:48:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:48:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:48:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:49:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:49:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:49:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-05T22:49:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:49:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:49:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:49:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:49:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-05T22:49:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-05T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:49:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-05T22:49:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-05T22:49:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:49:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-05T22:49:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2023-12-05T22:49:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:49:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:49:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:49:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:49:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:49:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:49:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2023-12-05T22:49:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:49:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:49:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:49:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:49:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:49:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:49:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-05T22:49:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-05T22:49:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:49:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:49:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:49:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-05T22:49:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:49:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-05T22:49:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2023-12-05T22:49:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T22:49:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-05T22:49:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:3 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 3 = cluster1:3 ] 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2023-12-05T22:49:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-05T22:49:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:49:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:49:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-05T22:49:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2023-12-05T22:49:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:49:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:49:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:49:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:49:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-05T22:49:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:49:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-05T22:49:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-05T22:49:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-05T22:49:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:49:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:49:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-05T22:49:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:49:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:49:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:49:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:49:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:49:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:49:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-12-05T22:49:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_44.status 2023-12-05T22:49:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_44.status 2023-12-05T22:49:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.status 2023-12-05T22:49:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_44.status 2023-12-05T22:49:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_44.status 2023-12-05T22:49:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-12-05T22:49:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-12-05T22:49:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 205 2023-12-05T22:49:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.status 2023-12-05T22:49:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.status 2023-12-05T22:49:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + rbd --cluster cluster2xmlstarlet snap sel -t list -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: --all mirror/image_44 --format xml 2023-12-05T22:49:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:49:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=205 2023-12-05T22:49:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:49:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-05T22:49:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=111 2023-12-05T22:49:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = 111 2023-12-05T22:49:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:49:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-05T22:49:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = 2023-12-05T22:49:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:49:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-05T22:49:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = 2023-12-05T22:49:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:49:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.status 2023-12-05T22:49:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-05T22:49:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=205 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = 205 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-12-05T22:49:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_44': 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12c2677f8d4c 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12c2677f8d4c 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:49:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:39:51 2023 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:39:51 2023 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:39:51 2023 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c768d0f5-279d-48dd-8b01-214e09ad3ae9 2023-12-05T22:49:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:49:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_44 snap 2023-12-05T22:49:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:49:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:49:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.export 2023-12-05T22:49:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.export 2023-12-05T22:49:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.export 2023-12-05T22:49:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.export 2023-12-05T22:49:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.export 2023-12-05T22:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.export 2023-12-05T22:49:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_44.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_44.export 2023-12-05T22:49:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2023-12-05T22:49:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_45 snap 2023-12-05T22:49:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:49:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-12-05T22:49:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-12-05T22:49:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:49:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-12-05T22:49:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:49:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:49:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ greplocal cluster=cluster1 -i 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:49:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:49:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-05T22:49:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:49:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo "state": "Replaying" 2023-12-05T22:49:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i Replaying 2023-12-05T22:49:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:49:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:49:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-12-05T22:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_45.status 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_45.status 2023-12-05T22:49:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.status 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_45.status 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_45.status 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.status 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-12-05T22:49:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 207 2023-12-05T22:49:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.status 2023-12-05T22:49:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.status 2023-12-05T22:49:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-12-05T22:49:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:49:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=207 2023-12-05T22:49:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:49:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.status 2023-12-05T22:49:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.status 2023-12-05T22:49:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-05T22:49:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=112 2023-12-05T22:49:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 207 = 112 2023-12-05T22:49:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:49:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.status 2023-12-05T22:49:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.status 2023-12-05T22:49:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-05T22:49:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 207 = 2023-12-05T22:49:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:49:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.status 2023-12-05T22:49:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.status 2023-12-05T22:49:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-05T22:49:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=207 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 207 = 207 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-12-05T22:49:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_45': 2023-12-05T22:49:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:49:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:49:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:49:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12cc9d6857c4 2023-12-05T22:49:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12cc9d6857c4 2023-12-05T22:49:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:49:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:49:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:49:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:49:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:05 2023 2023-12-05T22:49:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:05 2023 2023-12-05T22:49:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:05 2023 2023-12-05T22:49:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:49:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:49:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:49:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1f88b449-0d2e-442c-b595-3a88ba6e5d1a 2023-12-05T22:49:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:49:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_45 snap 2023-12-05T22:49:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:49:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:49:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.export 2023-12-05T22:49:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.export 2023-12-05T22:49:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.export 2023-12-05T22:49:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.export 2023-12-05T22:49:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.export 2023-12-05T22:49:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.export 2023-12-05T22:49:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_45.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_45.export 2023-12-05T22:49:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2023-12-05T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_46 snap 2023-12-05T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:49:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-12-05T22:49:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:49:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:49:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local instance=cluster1 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:49:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:49:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:49:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:49:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-12-05T22:49:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:49:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:49:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-12-05T22:49:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:49:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo "state": "Replaying"grep 2023-12-05T22:49:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: -i Replaying 2023-12-05T22:49:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_46.status 2023-12-05T22:49:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_46.status 2023-12-05T22:49:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.status 2023-12-05T22:49:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_46.status 2023-12-05T22:49:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_46.status 2023-12-05T22:49:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-12-05T22:49:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-12-05T22:49:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 209 2023-12-05T22:49:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.status 2023-12-05T22:49:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.status 2023-12-05T22:49:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-12-05T22:49:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:49:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=209 2023-12-05T22:49:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:49:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-05T22:49:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=113 2023-12-05T22:49:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = 113 2023-12-05T22:49:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-05T22:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:39.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:39.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = 2023-12-05T22:49:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:49:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-05T22:49:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = 2023-12-05T22:49:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:49:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.status 2023-12-05T22:49:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-05T22:49:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=209 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = 209 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_46': 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128929d5f057 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128929d5f057 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:49:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:14 2023 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:14 2023 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:14 2023 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4dff6617-95c0-4b5e-9363-8846654b6996 2023-12-05T22:49:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_46 snap 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.export 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.export 2023-12-05T22:49:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.export 2023-12-05T22:49:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.export 2023-12-05T22:49:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.export 2023-12-05T22:49:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.export 2023-12-05T22:49:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_46.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_46.export 2023-12-05T22:49:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2023-12-05T22:49:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_47 snap 2023-12-05T22:49:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:49:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-12-05T22:49:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-12-05T22:49:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:49:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-12-05T22:49:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:49:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:49:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons+ cluster1 rbd mirror status mirror/image_47 2023-12-05T22:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_instance=cluster1 2023-12-05T22:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state 2023-12-05T22:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:49:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:49:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:49:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:49:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:49:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-05T22:49:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:49:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:49:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:49:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-05T22:49:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-05T22:49:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-05T22:49:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:49:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:49:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:49:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:49:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:49:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-05T22:49:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-05T22:49:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:49:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:2 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 2 = cluster1:2 ] 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2023-12-05T22:49:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-05T22:49:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-05T22:49:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-12-05T22:49:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:49:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:49:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:49:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_47.status 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echosed -e cluster2-mirror-image_47.status s|[/ ]|_|g 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.status 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_47.status 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_47.status 2023-12-05T22:49:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.status 2023-12-05T22:49:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-12-05T22:49:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-12-05T22:49:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 211 2023-12-05T22:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.status 2023-12-05T22:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.status 2023-12-05T22:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-12-05T22:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:49:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=211 2023-12-05T22:49:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:49:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.status 2023-12-05T22:49:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.status 2023-12-05T22:49:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-05T22:49:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 211 = 2023-12-05T22:49:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:49:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.status 2023-12-05T22:49:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.status 2023-12-05T22:49:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-05T22:49:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 211 = 2023-12-05T22:49:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:49:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.status 2023-12-05T22:49:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.status 2023-12-05T22:49:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-05T22:49:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=211 2023-12-05T22:49:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 211 = 211 2023-12-05T22:49:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-12-05T22:49:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_47': 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12c2f15432ec 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12c2f15432ec 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:49:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:29 2023 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:29 2023 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:29 2023 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b3e4e6e0-3842-4e3a-8b00-addba2ae7c6f 2023-12-05T22:49:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:49:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_47 snap 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.export 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.export 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.export 2023-12-05T22:49:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.export 2023-12-05T22:49:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.export 2023-12-05T22:49:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:49:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.export 2023-12-05T22:49:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_47.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_47.export 2023-12-05T22:49:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2023-12-05T22:49:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_48 snap 2023-12-05T22:49:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:49:51.588 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.39324022034310313) > weight (0.25) 2023-12-05T22:49:51.588 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6329327220149606) > weight (0.25) 2023-12-05T22:49:51.588 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-05T22:49:51.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-05T22:49:51.589 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-05T22:49:51.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:49:51.594+0000 7fa87ceec640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 53162) UID: 1000 2023-12-05T22:49:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-12-05T22:49:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-12-05T22:49:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-12-05T22:49:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:49:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:49:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:49:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-12-05T22:49:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local+ cluster_instance=cluster1 2023-12-05T22:49:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ localgrep -i cluster=cluster1 state 2023-12-05T22:49:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance=cluster1 2023-12-05T22:49:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:49:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:49:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:49:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:49:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-12-05T22:49:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:49:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:49:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:49:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:49:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:49:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:49:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:49:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-12-05T22:49:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:49:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:49:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:49:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:49:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_48.status 2023-12-05T22:49:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_48.status 2023-12-05T22:49:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.status 2023-12-05T22:49:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_48.status 2023-12-05T22:49:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_48.status 2023-12-05T22:49:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:49:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-12-05T22:49:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-12-05T22:49:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 213 2023-12-05T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.status 2023-12-05T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.status 2023-12-05T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-12-05T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:49:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=213 2023-12-05T22:49:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:49:55.589 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-05T22:49:55.589 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:49:55.590 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-05T22:49:55.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-05T22:49:55.590 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-05T22:49:55.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-05T22:49:55.593 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-05T22:49:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-05T22:49:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=115 2023-12-05T22:49:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = 115 2023-12-05T22:49:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-05T22:49:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = 2023-12-05T22:49:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:49:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-05T22:49:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:49:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:49:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:49:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = 2023-12-05T22:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2023-12-05T22:49:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.status 2023-12-05T22:49:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-05T22:49:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:49:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=213 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = 213 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:49:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-12-05T22:49:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_48': 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12cc524ac828 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12cc524ac828 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:38 2023 2023-12-05T22:49:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:38 2023 2023-12-05T22:49:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:38 2023 2023-12-05T22:49:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:49:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:49:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:49:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4c38d01c-4d6b-4116-a8ce-10e93c8bf38b 2023-12-05T22:49:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_48 snap 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.export 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.export 2023-12-05T22:49:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.export 2023-12-05T22:49:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.export 2023-12-05T22:50:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:50:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.export 2023-12-05T22:50:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:50:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.export 2023-12-05T22:50:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_48.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_48.export 2023-12-05T22:50:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2023-12-05T22:50:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_49 snap 2023-12-05T22:50:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:50:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:50:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:50:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:50:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-12-05T22:50:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local cluster_instance=cluster1 2023-12-05T22:50:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:50:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i state+ 2023-12-05T22:50:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:local instance=cluster1 2023-12-05T22:50:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:50:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:50:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:50:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-12-05T22:50:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:50:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:50:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:50:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:50:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:50:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-12-05T22:50:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:50:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:50:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:50:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:50:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-12-05T22:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-12-05T22:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_49.status 2023-12-05T22:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + echo cluster2-mirror-image_49.status 2023-12-05T22:50:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -e s|[/ ]|_|g 2023-12-05T22:50:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.status 2023-12-05T22:50:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_49.status 2023-12-05T22:50:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_49.status 2023-12-05T22:50:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:50:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.status 2023-12-05T22:50:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-12-05T22:50:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-12-05T22:50:04.397 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3959196175370593) > weight (0.25) 2023-12-05T22:50:04.397 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6593533186440724) > weight (0.25) 2023-12-05T22:50:04.397 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-05T22:50:04.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-05T22:50:04.398 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-05T22:50:04.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:50:04.406+0000 7f36e1d5f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 64696) UID: 1000 2023-12-05T22:50:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 215 2023-12-05T22:50:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.status 2023-12-05T22:50:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.status 2023-12-05T22:50:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-12-05T22:50:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:50:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=215 2023-12-05T22:50:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:50:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:50:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.status 2023-12-05T22:50:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.status 2023-12-05T22:50:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-05T22:50:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:50:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:50:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:50:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:50:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:50:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:50:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 215 = 2023-12-05T22:50:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:50:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.status 2023-12-05T22:50:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.status 2023-12-05T22:50:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-05T22:50:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:05.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:50:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:50:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:50:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:50:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:50:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:50:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 215 = 2023-12-05T22:50:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:50:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.status 2023-12-05T22:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.status 2023-12-05T22:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-05T22:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=215 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 215 = 215 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:50:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:50:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_49': 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1140cfee0ab 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1140cfee0ab 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:40:52 2023 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:40:52 2023 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:40:52 2023 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f77f683c-79d4-42df-af67-46a03922b77d 2023-12-05T22:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:50:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_49 snap 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.export 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.export 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.export 2023-12-05T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.export 2023-12-05T22:50:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:50:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.export 2023-12-05T22:50:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:50:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.export 2023-12-05T22:50:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_49.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_49.export 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_50 snap 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:50:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-12-05T22:50:09.398 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-05T22:50:09.399 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:50:09.399 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-05T22:50:09.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-05T22:50:09.399 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-05T22:50:09.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-05T22:50:09.402 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-05T22:50:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-12-05T22:50:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:50:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:50:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-12-05T22:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2023-12-05T22:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2023-12-05T22:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2023-12-05T22:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local+ cluster_instance=cluster1 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -i+ statelocal 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: instance=cluster1 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loop_instance 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 != cluster1 ] 2023-12-05T22:50:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T22:50:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:50:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:0 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 = cluster1:0 ] 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:50:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-05T22:50:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:50:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:50:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-05T22:50:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-12-05T22:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-05T22:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2023-12-05T22:50:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:50:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1:1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2023-12-05T22:50:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=local_cluster 2023-12-05T22:50:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:50:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:50:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:50:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 = cluster1:1 ] 2023-12-05T22:50:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval local_cluster=cluster1 2023-12-05T22:50:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local_cluster=cluster1 2023-12-05T22:50:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2023-12-05T22:50:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2023-12-05T22:50:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-05T22:50:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:50:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:50:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-05T22:50:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-12-05T22:50:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result= "state": "Replaying" 2023-12-05T22:50:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo "state": "Replaying" 2023-12-05T22:50:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2023-12-05T22:50:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2023-12-05T22:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2023-12-05T22:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2023-12-05T22:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = journal ] 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ snapshot = snapshot ] 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster2-mirror-image_50.status 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster2-mirror-image_50.status 2023-12-05T22:50:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:50:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.status 2023-12-05T22:50:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mkfname cluster1-mirror-image_50.status 2023-12-05T22:50:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo cluster1-mirror-image_50.status 2023-12-05T22:50:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -e s|[/ ]|_|g 2023-12-05T22:50:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.status 2023-12-05T22:50:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-12-05T22:50:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-12-05T22:50:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 217 2023-12-05T22:50:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.status 2023-12-05T22:50:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.status 2023-12-05T22:50:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-12-05T22:50:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-05T22:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=217 2023-12-05T22:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2023-12-05T22:50:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.status 2023-12-05T22:50:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.status 2023-12-05T22:50:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-05T22:50:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:50:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=117 2023-12-05T22:50:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 217 = 117 2023-12-05T22:50:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2023-12-05T22:50:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.status 2023-12-05T22:50:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.status 2023-12-05T22:50:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-05T22:50:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:50:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:50:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2023-12-05T22:50:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:50:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2023-12-05T22:50:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2023-12-05T22:50:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 217 = 2023-12-05T22:50:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2023-12-05T22:50:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.status 2023-12-05T22:50:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.status 2023-12-05T22:50:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-05T22:50:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-05T22:50:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-05T22:50:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=217 2023-12-05T22:50:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 217 = 217 2023-12-05T22:50:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-12-05T22:50:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:50:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:50:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:50:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:50:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-12-05T22:50:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_50': 2023-12-05T22:50:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2023-12-05T22:50:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:50:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1289a338dc8e 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1289a338dc8e 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:41:00 2023 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:41:00 2023 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:41:00 2023 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: dab6d77c-da14-4bca-b431-d1248425b922 2023-12-05T22:50:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2023-12-05T22:50:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_50 snap 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.export 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.export 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.export 2023-12-05T22:50:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.export 2023-12-05T22:50:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:50:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.export 2023-12-05T22:50:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-05T22:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.export 2023-12-05T22:50:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.yK28jjcyo4/cluster2-mirror-image_50.export /tmp/tmp.yK28jjcyo4/cluster1-mirror-image_50.export 2023-12-05T22:50:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog TEST: delete many images 2023-12-05T22:50:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + teedate -a +%F %T /tmp/tmp.yK28jjcyo4/rbd-mirror.test.log 2023-12-05T22:50:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T22:50:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2023-12-05 22:50:15 TEST: delete many images 2023-12-05T22:50:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2023-12-05 22:50:15 TEST: delete many images 2023-12-05T22:50:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 50 2023-12-05T22:50:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2023-12-05T22:50:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-12-05T22:50:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:50:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-12-05T22:50:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-05T22:50:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:50:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_1 2023-12-05T22:50:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:50:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-05T22:50:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-05T22:50:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:15.937+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e287040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:15.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:15.937+0000 7f86718ea640 -1 rbd::mirror::ImageReplayer: 0x55f03e81ec80 [3/6dc8b84c-1b31-4882-988d-5f4b4f42faf2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:17.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2023-12-05T22:50:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-12-05T22:50:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:50:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-12-05T22:50:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-05T22:50:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:50:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_2 2023-12-05T22:50:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:50:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-05T22:50:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-05T22:50:18.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:50:18.801+0000 7fdfae110640 -1 rbd::mirror::ImageMap: 0x55e02607f8c0 handle_peer_ack_remove: failed to remove global_image_id=9b0c0c99-2835-43a4-88c9-c3397c0e79b3 2023-12-05T22:50:18.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:18.821+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb6cb1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:18.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:18.821+0000 7fcd0f482640 -1 rbd::mirror::ImageReplayer: 0x561ddae20f00 [3/9b0c0c99-2835-43a4-88c9-c3397c0e79b3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-12-05T22:50:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-05T22:50:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:50:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_3 2023-12-05T22:50:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:50:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-05T22:50:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-05T22:50:21.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:21.637+0000 7fcd08c75640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddbc0bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:21.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:21.637+0000 7fcd0ec81640 -1 rbd::mirror::ImageReplayer: 0x561ddae20500 [3/09ba1f5d-4c90-48e3-a545-11284c264b5a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2023-12-05T22:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-12-05T22:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-12-05T22:50:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-05T22:50:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:50:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_4 2023-12-05T22:50:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:50:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-05T22:50:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-05T22:50:24.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:24.661+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb66ed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:24.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:24.661+0000 7fcd0ec81640 -1 rbd::mirror::ImageReplayer: 0x561ddae21400 [3/aa119937-ee52-43d0-81c6-311c63b77e8f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2023-12-05T22:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-12-05T22:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-12-05T22:50:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-05T22:50:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:50:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_5 2023-12-05T22:50:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:50:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-05T22:50:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-05T22:50:27.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:50:27.617+0000 7f051ba4b640 -1 rbd::mirror::ImageReplayer: 0x564eedea4780 [3/23aac29b-db31-4b69-bed6-9ec5773f48c0] start: already running 2023-12-05T22:50:27.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:27.685+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e229d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:27.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:27.685+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03e81e500 [3/23aac29b-db31-4b69-bed6-9ec5773f48c0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2023-12-05T22:50:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-12-05T22:50:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:50:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-12-05T22:50:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-05T22:50:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:50:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_6 2023-12-05T22:50:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:50:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-05T22:50:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-05T22:50:30.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:50:30.733+0000 7f800bfc8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556400cd5a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:30.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:50:30.733+0000 7f80127d5640 -1 rbd::mirror::ImageReplayer: 0x556400411180 [3/ee1e6e37-e0d9-4a81-a633-71147ddff486] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-12-05T22:50:32.782 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:50:32.778+0000 7f1618813640 -1 reset not still connected to 0x558e250e52b0 2023-12-05T22:50:32.782 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:50:32.778+0000 7f1618813640 -1 reset not still connected to 0x558e250e55f0 2023-12-05T22:50:32.782 INFO:tasks.ceph.cluster1.osd.2.smithi062.stderr:2023-12-05T22:50:32.778+0000 7f1618813640 -1 reset not still connected to 0x558e2566b5f0 2023-12-05T22:50:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-05T22:50:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:50:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_7 2023-12-05T22:50:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:50:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-05T22:50:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-05T22:50:33.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:33.733+0000 7fcd08c75640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb670680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:33.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:33.733+0000 7fcd0ec81640 -1 rbd::mirror::ImageReplayer: 0x561ddae20280 [3/01c4381c-2953-4889-8d37-61725def0e37] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2023-12-05T22:50:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-12-05T22:50:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:50:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-12-05T22:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-05T22:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_8 2023-12-05T22:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-05T22:50:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-05T22:50:36.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:36.657+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2d89c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:36.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:36.657+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03e81ef00 [3/5e8d2584-d40c-4e8c-bda1-2d1342a10e52] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-12-05T22:50:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-05T22:50:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:50:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_9 2023-12-05T22:50:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:50:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-05T22:50:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-05T22:50:39.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:39.792+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb66eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:39.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:39.792+0000 7fcd0f482640 -1 rbd::mirror::ImageReplayer: 0x561ddae21b80 [3/d0e14dbe-197f-4ab0-8cb5-c1acd3e96372] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:40.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:40.870+0000 7efd5eea3640 -1 reset not still connected to 0x55aea7bcdad0 2023-12-05T22:50:40.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:40.870+0000 7efd5eea3640 -1 reset not still connected to 0x55aea8ffcd00 2023-12-05T22:50:40.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:40.870+0000 7efd5eea3640 -1 reset not still connected to 0x55aea9ed65b0 2023-12-05T22:50:40.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:40.870+0000 7efd5eea3640 -1 reset not still connected to 0x55aea9ed6680 2023-12-05T22:50:40.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:40.870+0000 7efd5eea3640 -1 reset not still connected to 0x55aeaab63ee0 2023-12-05T22:50:40.875 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:40.870+0000 7efd5eea3640 -1 reset not still connected to 0x55aead0d5ad0 2023-12-05T22:50:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2023-12-05T22:50:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-12-05T22:50:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:50:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-12-05T22:50:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_10 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:50:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-05T22:50:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-05T22:50:42.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:50:42.856+0000 7f800b7c7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556400cd51e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:42.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:50:42.856+0000 7f80127d5640 -1 rbd::mirror::ImageReplayer: 0x556400410280 [3/0042551e-e667-4458-be32-9094ff2c9220] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-12-05T22:50:44.972 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:44.970+0000 7efd5eea3640 -1 reset not still connected to 0x55aeaab63ee0 2023-12-05T22:50:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_11 2023-12-05T22:50:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:50:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_11 2023-12-05T22:50:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:50:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-12-05T22:50:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-12-05T22:50:45.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:50:45.852+0000 7f800b7c7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556401294000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:45.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:50:45.852+0000 7f80127d5640 -1 rbd::mirror::ImageReplayer: 0x556400410500 [3/08d57849-e22c-4949-ad8e-26b3cce133fe] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2023-12-05T22:50:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-12-05T22:50:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:50:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-12-05T22:50:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_12 2023-12-05T22:50:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:50:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_12 2023-12-05T22:50:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:50:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-12-05T22:50:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-12-05T22:50:48.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:48.660+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb6cb1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:48.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:50:48.660+0000 7fcd0ec81640 -1 rbd::mirror::ImageReplayer: 0x561ddae20f00 [3/73016fb1-14a6-445e-a3b2-f2065161bec3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-12-05T22:50:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_13 2023-12-05T22:50:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:50:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_13 2023-12-05T22:50:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:50:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-12-05T22:50:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-12-05T22:50:51.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:50:51.676+0000 7f5b9fa14640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb95392340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:51.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:50:51.676+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb94e84000 [3/d5db6364-6a7e-48b6-a251-4e792c42f0d9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2023-12-05T22:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-12-05T22:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-12-05T22:50:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_14 2023-12-05T22:50:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:50:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_14 2023-12-05T22:50:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:50:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-12-05T22:50:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-12-05T22:50:54.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:50:54.732+0000 7f5b9fa14640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb954f6340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:54.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:50:54.732+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447a280 [3/c9eb233a-812a-48e1-98fd-e7779e2fce61] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-12-05T22:50:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:50:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_15 2023-12-05T22:50:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:50:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:50:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_15 2023-12-05T22:50:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:50:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-12-05T22:50:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-12-05T22:50:57.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:57.716+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2d8b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:50:57.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:50:57.716+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03e81fb80 [3/b9b1e588-b890-4a39-a107-1d78081756ec] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:50:59.568 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:59.565+0000 7efd5eea3640 -1 reset not still connected to 0x55aeaae6fee0 2023-12-05T22:50:59.569 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:50:59.565+0000 7efd5eea3640 -1 reset not still connected to 0x55aead1352b0 2023-12-05T22:50:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:50:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:50:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2023-12-05T22:50:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-12-05T22:50:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:50:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:50:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:50:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:50:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-12-05T22:51:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_16 2023-12-05T22:51:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:51:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_16 2023-12-05T22:51:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:51:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-12-05T22:51:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-12-05T22:51:00.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:51:00.780+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb8ec9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:00.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:51:00.780+0000 7fcd0f482640 -1 rbd::mirror::ImageReplayer: 0x561ddae21680 [3/b3ff1efc-2dee-4937-993c-fb819906b94f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2023-12-05T22:51:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-12-05T22:51:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:51:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-12-05T22:51:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_17 2023-12-05T22:51:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:51:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_17 2023-12-05T22:51:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:51:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-12-05T22:51:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-12-05T22:51:03.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:51:03.788+0000 7fcd08c75640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb8ecb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:03.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:51:03.788+0000 7fcd0f482640 -1 rbd::mirror::ImageReplayer: 0x561ddae21180 [3/9f56dd88-b64c-4071-b253-84119be63f20] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2023-12-05T22:51:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-12-05T22:51:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:51:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-12-05T22:51:05.931 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:05.933+0000 7efd5eea3640 -1 reset not still connected to 0x55aea9ed6680 2023-12-05T22:51:05.931 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:05.933+0000 7efd5eea3640 -1 reset not still connected to 0x55aeaab63ee0 2023-12-05T22:51:05.931 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:05.933+0000 7efd5eea3640 -1 reset not still connected to 0x55aead0d5ad0 2023-12-05T22:51:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_18 2023-12-05T22:51:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:51:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_18 2023-12-05T22:51:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:51:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-12-05T22:51:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-12-05T22:51:06.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:51:06.700+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2d8d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:06.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:51:06.700+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03e81f180 [3/a058c447-ff5e-49d0-9009-af35f59afbb5] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2023-12-05T22:51:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-12-05T22:51:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:51:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-12-05T22:51:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_19 2023-12-05T22:51:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_19 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:51:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-12-05T22:51:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-12-05T22:51:09.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:51:09.847+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2d8ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:09.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:51:09.847+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03d986280 [3/e72c6cbf-3629-4ab9-855f-c2cdd06d81a7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:10.403 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.682492983162856) > weight (0.25) 2023-12-05T22:51:10.403 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9380671390639207) > weight (0.25) 2023-12-05T22:51:10.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.44631077404007136) > weight (0.25) 2023-12-05T22:51:10.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8659286220839619) > weight (0.25) 2023-12-05T22:51:10.404 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-05T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2023-12-05T22:51:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-12-05T22:51:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:51:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-12-05T22:51:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_20 2023-12-05T22:51:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:51:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_20 2023-12-05T22:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-12-05T22:51:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-12-05T22:51:12.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:51:12.887+0000 7fcd08c75640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb8ecd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:12.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:51:12.887+0000 7fcd0f482640 -1 rbd::mirror::ImageReplayer: 0x561ddae20780 [3/1533eb9c-c4ac-468b-9060-8f72f4f33772] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-12-05T22:51:14.942 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:14.945+0000 7efd5eea3640 -1 reset not still connected to 0x55aead0d5ad0 2023-12-05T22:51:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_21 2023-12-05T22:51:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:51:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_21 2023-12-05T22:51:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:51:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-12-05T22:51:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-12-05T22:51:15.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:15.887+0000 7f800bfc8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5564012941a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:15.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:15.887+0000 7f80127d5640 -1 rbd::mirror::ImageReplayer: 0x556400410780 [3/18f502e6-d626-414b-b187-fc0882719611] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-12-05T22:51:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_22 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_22 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:51:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-12-05T22:51:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-12-05T22:51:18.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:18.703+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb95392ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:18.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:18.703+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb9447a500 [3/e296de3e-e023-46cc-9b13-df738af23268] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2023-12-05T22:51:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-12-05T22:51:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:51:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-12-05T22:51:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_23 2023-12-05T22:51:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:51:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_23 2023-12-05T22:51:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:51:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-12-05T22:51:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-12-05T22:51:21.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:51:21.691+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2d9040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:21.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:51:21.691+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03d986a00 [3/e9b3d2ae-1928-4b14-803b-779daf73fd91] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2023-12-05T22:51:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-12-05T22:51:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:51:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-12-05T22:51:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_24 2023-12-05T22:51:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:51:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_24 2023-12-05T22:51:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:51:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-12-05T22:51:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-12-05T22:51:24.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:24.747+0000 7f5b9fa14640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb954f7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:24.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:24.747+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447a780 [3/17202bda-24c3-40df-b59d-9904509b75df] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2023-12-05T22:51:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-12-05T22:51:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:51:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-12-05T22:51:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_25 2023-12-05T22:51:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:51:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_25 2023-12-05T22:51:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:51:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-12-05T22:51:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-12-05T22:51:27.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:27.831+0000 7f5b9fa14640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb947d7040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:27.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:27.831+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447b400 [3/4ff38d91-f124-4824-8d8b-0b2a3eb228a9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-12-05T22:51:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_26 2023-12-05T22:51:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:51:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_26 2023-12-05T22:51:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:51:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-12-05T22:51:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-12-05T22:51:30.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:30.827+0000 7f800b7c7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556401294340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:30.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:30.827+0000 7f80127d5640 -1 rbd::mirror::ImageReplayer: 0x556400411400 [3/119a68f7-e780-449a-acf3-897b2c081275] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:32.492 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:51:32.495+0000 7fdd570d4640 -1 reset not still connected to 0x559d3e8f16c0 2023-12-05T22:51:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-12-05T22:51:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_27 2023-12-05T22:51:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:51:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_27 2023-12-05T22:51:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:51:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-12-05T22:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-12-05T22:51:33.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:33.823+0000 7f800bfc8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556400c971e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:33.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:33.823+0000 7f8011fd4640 -1 rbd::mirror::ImageReplayer: 0x556400402000 [3/d5a19c81-aa40-411c-8f9d-e68f5717dee7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2023-12-05T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-12-05T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:51:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-12-05T22:51:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_28 2023-12-05T22:51:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:51:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_28 2023-12-05T22:51:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:51:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-12-05T22:51:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-12-05T22:51:36.600 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:36.601+0000 7efd5eea3640 -1 reset not still connected to 0x55aea8ffcd00 2023-12-05T22:51:36.601 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:36.601+0000 7efd5eea3640 -1 reset not still connected to 0x55aea9ed65b0 2023-12-05T22:51:36.602 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:36.601+0000 7efd5eea3640 -1 reset not still connected to 0x55aea9ed6680 2023-12-05T22:51:36.602 INFO:tasks.ceph.cluster1.osd.0.smithi062.stderr:2023-12-05T22:51:36.601+0000 7efd5eea3640 -1 reset not still connected to 0x55aead0d5ad0 2023-12-05T22:51:36.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:36.687+0000 7f800bfc8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5564012944e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:36.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:36.687+0000 7f80127d5640 -1 rbd::mirror::ImageReplayer: 0x556400410a00 [3/188e2fd3-baca-493b-8be5-724525683def] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2023-12-05T22:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-12-05T22:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:51:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-12-05T22:51:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_29 2023-12-05T22:51:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:51:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_29 2023-12-05T22:51:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:51:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-12-05T22:51:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-12-05T22:51:39.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:39.847+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb94a9e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:39.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:39.847+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447bb80 [3/4f42a30e-47bf-4a43-ad22-e89c1569217d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2023-12-05T22:51:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-12-05T22:51:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:51:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-12-05T22:51:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_30 2023-12-05T22:51:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:51:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_30 2023-12-05T22:51:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:51:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-12-05T22:51:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-12-05T22:51:42.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:42.914+0000 7f800bfc8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556400f4b860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:42.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:42.914+0000 7f8011fd4640 -1 rbd::mirror::ImageReplayer: 0x556400411900 [3/607abc1b-0895-4b2d-abb8-f771bc5c407f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2023-12-05T22:51:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-12-05T22:51:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:51:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-12-05T22:51:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_31 2023-12-05T22:51:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:51:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_31 2023-12-05T22:51:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:51:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-12-05T22:51:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-12-05T22:51:45.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:45.902+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb95393860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:45.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:45.902+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb9447ac80 [3/36786ed1-b139-4c21-bcaa-0cacd1cad4dc] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-12-05T22:51:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:48.392 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:51:48.394+0000 7fdd570d4640 -1 reset not still connected to 0x559d3de2e4e0 2023-12-05T22:51:48.392 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:51:48.394+0000 7fdd570d4640 -1 reset not still connected to 0x559d3deb9040 2023-12-05T22:51:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_32 2023-12-05T22:51:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:51:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_32 2023-12-05T22:51:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:51:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-12-05T22:51:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-12-05T22:51:48.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:48.770+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb947dcea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:48.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:48.770+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447b680 [3/726117d0-c60f-4f67-b079-12119b0741a6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-12-05T22:51:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_33 2023-12-05T22:51:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:51:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_33 2023-12-05T22:51:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:51:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-12-05T22:51:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-12-05T22:51:51.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:51.746+0000 7f800b7c7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556400f4a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:51.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:51.746+0000 7f8011fd4640 -1 rbd::mirror::ImageReplayer: 0x556400411b80 [3/80cd4891-b460-4331-9eae-304b773fde31] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:52.593 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5418217111997485) > weight (0.25) 2023-12-05T22:51:52.594 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:51:52.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:51:52.594 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-05T22:51:52.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:51:52.598+0000 7f8012fd6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 61919) UID: 1000 2023-12-05T22:51:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2023-12-05T22:51:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-12-05T22:51:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:51:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-12-05T22:51:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_34 2023-12-05T22:51:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:51:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_34 2023-12-05T22:51:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:51:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-12-05T22:51:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-12-05T22:51:54.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:54.794+0000 7f5b9fa14640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb95393a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:51:54.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:54.794+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb9447aa00 [3/c95cf52a-6e99-4b13-94b9-00efacfbc9bf] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:51:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2023-12-05T22:51:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-12-05T22:51:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:51:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-12-05T22:51:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:51:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_35 2023-12-05T22:51:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:51:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:51:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_35 2023-12-05T22:51:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:51:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-12-05T22:51:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-12-05T22:51:58.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:51:58.034+0000 7f5ba5a20640 -1 rbd::mirror::ImageMap: 0x55cb94b11320 handle_peer_ack_remove: failed to remove global_image_id=1d6600a5-89f1-4e94-812c-f425e83f649a 2023-12-05T22:51:59.595 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:51:59.595 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:51:59.596 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:51:59.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:51:59.597 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:51:59.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:51:59.601 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-05T22:51:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:51:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:51:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2023-12-05T22:51:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-12-05T22:51:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:51:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:51:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:51:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:51:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-12-05T22:52:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_36 2023-12-05T22:52:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:52:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_36 2023-12-05T22:52:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:52:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-12-05T22:52:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-12-05T22:52:00.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:00.838+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e287ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:00.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:00.838+0000 7f86718ea640 -1 rbd::mirror::ImageReplayer: 0x55f03e81ea00 [3/4f5aa172-1810-4fc1-84bc-bfbb3a129dcd] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2023-12-05T22:52:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-12-05T22:52:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:52:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-12-05T22:52:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_37 2023-12-05T22:52:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:52:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_37 2023-12-05T22:52:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:52:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-12-05T22:52:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-12-05T22:52:03.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:03.838+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb94f9d6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:03.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:03.838+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447b900 [3/21086603-d365-40b9-9a1b-a87abe17542f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-12-05T22:52:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_38 2023-12-05T22:52:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:52:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_38 2023-12-05T22:52:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:52:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-12-05T22:52:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-12-05T22:52:06.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:06.734+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb954dc9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:06.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:06.734+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb94e84f00 [3/ffdb2cf0-ec49-4f59-9404-9cf65a1b49e0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-12-05T22:52:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_39 2023-12-05T22:52:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:52:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_39 2023-12-05T22:52:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:52:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-12-05T22:52:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-12-05T22:52:09.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:09.902+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e079d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:09.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:09.902+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03e81f680 [3/b11ea2a8-7502-44da-ba95-ff96ab7a913f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-12-05T22:52:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_40 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_40 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-12-05T22:52:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-12-05T22:52:12.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:52:12.954+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb8ed520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:12.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:52:12.954+0000 7fcd0f482640 -1 rbd::mirror::ImageReplayer: 0x561ddae20a00 [3/8480990e-2ffd-4b7c-89de-bb66a57203df] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2023-12-05T22:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-12-05T22:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-12-05T22:52:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_41 2023-12-05T22:52:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:52:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_41 2023-12-05T22:52:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:52:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-12-05T22:52:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-12-05T22:52:15.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:52:15.957+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb8ec1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:15.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:52:15.957+0000 7fcd0ec81640 -1 rbd::mirror::ImageReplayer: 0x561ddb7b0000 [3/f48ece28-d198-4bf2-878b-dd40720a4c29] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2023-12-05T22:52:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-12-05T22:52:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:52:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-12-05T22:52:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_42 2023-12-05T22:52:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:52:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_42 2023-12-05T22:52:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:52:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-12-05T22:52:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-12-05T22:52:18.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:18.741+0000 7f5ba5a20640 -1 rbd::mirror::ImageMap: 0x55cb94b11320 handle_peer_ack_remove: failed to remove global_image_id=50eb144e-0936-49e6-8545-fb513c5ca376 2023-12-05T22:52:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2023-12-05T22:52:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-12-05T22:52:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:52:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-12-05T22:52:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:21.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7030943623426025) > weight (0.25) 2023-12-05T22:52:21.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9604616654731267) > weight (0.25) 2023-12-05T22:52:21.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7824804070483065) > weight (0.25) 2023-12-05T22:52:21.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7780541136643353) > weight (0.25) 2023-12-05T22:52:21.405 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2023-12-05T22:52:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_43 2023-12-05T22:52:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:52:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_43 2023-12-05T22:52:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:52:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-12-05T22:52:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-12-05T22:52:21.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:21.713+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e273d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:21.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:21.713+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03e81f400 [3/a0acf697-0cfe-4b8b-880e-ea4251b3ce8c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-12-05T22:52:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_44 2023-12-05T22:52:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_44 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:52:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-12-05T22:52:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-12-05T22:52:24.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:24.829+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e31cd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:24.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:24.829+0000 7f86718ea640 -1 rbd::mirror::ImageReplayer: 0x55f03e81f900 [3/c768d0f5-279d-48dd-8b01-214e09ad3ae9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2023-12-05T22:52:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-12-05T22:52:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:52:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-12-05T22:52:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_45 2023-12-05T22:52:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:52:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_45 2023-12-05T22:52:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:52:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-12-05T22:52:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-12-05T22:52:27.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:52:27.845+0000 7fe3461e7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b15906aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:27.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:52:27.845+0000 7fe34c1f3640 -1 rbd::mirror::ImageReplayer: 0x55b15934c500 [3/1f88b449-0d2e-442c-b595-3a88ba6e5d1a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-12-05T22:52:29.835 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:52:29.841+0000 7fe8efb13640 -1 reset not still connected to 0x55b5fad756c0 2023-12-05T22:52:29.837 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:52:29.841+0000 7fe8efb13640 -1 reset not still connected to 0x55b5fb98a340 2023-12-05T22:52:29.838 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:52:29.841+0000 7fe8efb13640 -1 reset not still connected to 0x55b6007ad5f0 2023-12-05T22:52:29.838 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:52:29.841+0000 7fe8efb13640 -1 reset not still connected to 0x55b629f745b0 2023-12-05T22:52:29.838 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:52:29.845+0000 7fe8efb13640 -1 reset not still connected to 0x55b62c919860 2023-12-05T22:52:29.839 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:52:29.845+0000 7fe8efb13640 -1 reset not still connected to 0x55b62f8d6a90 2023-12-05T22:52:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_46 2023-12-05T22:52:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:52:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_46 2023-12-05T22:52:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:52:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-12-05T22:52:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-12-05T22:52:30.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:30.853+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb94fd3ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:30.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:30.853+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb94e84f00 [3/4dff6617-95c0-4b5e-9363-8846654b6996] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2023-12-05T22:52:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-12-05T22:52:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:52:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-12-05T22:52:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_47 2023-12-05T22:52:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:52:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_47 2023-12-05T22:52:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:52:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-12-05T22:52:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-12-05T22:52:33.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:52:33.837+0000 7fcd08474640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561ddb989ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:33.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T22:52:33.837+0000 7fcd0f482640 -1 rbd::mirror::ImageReplayer: 0x561ddae21900 [3/b3e4e6e0-3842-4e3a-8b00-addba2ae7c6f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2023-12-05T22:52:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-12-05T22:52:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:52:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-12-05T22:52:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_48 2023-12-05T22:52:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:52:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_48 2023-12-05T22:52:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:52:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-12-05T22:52:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-12-05T22:52:36.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:36.765+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2d8d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:36.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:36.765+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03e81e780 [3/4c38d01c-4d6b-4116-a8ce-10e93c8bf38b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2023-12-05T22:52:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-12-05T22:52:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:52:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-12-05T22:52:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_49 2023-12-05T22:52:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:52:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_49 2023-12-05T22:52:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:52:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-12-05T22:52:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-12-05T22:52:39.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:39.933+0000 7f5b9fa14640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb954dda00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:39.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:52:39.933+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb94e84780 [3/f77f683c-79d4-42df-af67-46a03922b77d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2023-12-05T22:52:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-12-05T22:52:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:52:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2023-12-05T22:52:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-12-05T22:52:41.954 INFO:tasks.ceph.cluster1.osd.1.smithi062.stderr:2023-12-05T22:52:41.948+0000 7f4b0ce40640 -1 reset not still connected to 0x55b1196c31e0 2023-12-05T22:52:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2023-12-05T22:52:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_50 2023-12-05T22:52:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:52:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:52:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_50 2023-12-05T22:52:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:52:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-12-05T22:52:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-12-05T22:52:42.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:42.905+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2d8ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:52:42.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:52:42.905+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03d986000 [3/dab6d77c-da14-4bca-b431-d1248425b922] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:52:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-05T22:52:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog TEST: image deletions should propagate 2023-12-05T22:52:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.yK28jjcyo4/rbd-mirror.test.log 2023-12-05T22:52:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ date +%F %T 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2023-12-05 22:52:44 TEST: image deletions should propagate 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2023-12-05 22:52:44 TEST: image deletions should propagate 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_count=0 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local count 2023-12-05T22:52:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_count=0 2023-12-05T22:52:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2023-12-05T22:52:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 40 2023-12-05T22:52:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2023-12-05T22:52:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-05T22:52:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep images: 2023-12-05T22:52:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cut -d -f 2 2023-12-05T22:52:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:52:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count=0 2023-12-05T22:52:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2023-12-05T22:52:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-12-05T22:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state 2023-12-05T22:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 40 2023-12-05T22:52:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2023-12-05T22:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-05T22:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep image health: 2023-12-05T22:52:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cut -d -f 3 2023-12-05T22:52:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2023-12-05T22:52:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=OK 2023-12-05T22:52:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = ERROR 2023-12-05T22:52:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = OK 2023-12-05T22:52:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 50 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_1 2023-12-05T22:52:44.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:44.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:44.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:52:44.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-05T22:52:44.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-05T22:52:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_1 deleted 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_1 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-05T22:52:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_2 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-05T22:52:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-05T22:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_2 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-05T22:52:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-05T22:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_3 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:52:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-05T22:52:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-05T22:52:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-12-05T22:52:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:52:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_3 2023-12-05T22:52:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:52:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-05T22:52:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-05T22:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_4 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-05T22:52:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-05T22:52:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_4 deleted 2023-12-05T22:52:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:52:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_4 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-05T22:52:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-05T22:52:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2023-12-05T22:52:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_5 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-05T22:52:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-05T22:52:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_5 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-05T22:52:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:52:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:52:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-05T22:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:52:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:52:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:52:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-05T22:52:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_7 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:52:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-05T22:52:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-05T22:52:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_7 2023-12-05T22:52:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:52:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-05T22:52:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-05T22:52:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_8 2023-12-05T22:52:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:52:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-05T22:52:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-05T22:52:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-12-05T22:52:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:52:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_8 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-05T22:52:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_9 2023-12-05T22:52:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:52:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-05T22:52:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-05T22:52:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_9 deleted 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_9 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-05T22:52:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-05T22:52:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_10 2023-12-05T22:52:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:52:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-05T22:52:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-05T22:52:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_10 deleted 2023-12-05T22:52:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_10 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-05T22:52:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-05T22:52:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_11 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-12-05T22:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-05T22:52:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_11 deleted 2023-12-05T22:52:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_11 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2023-12-05T22:52:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-12-05T22:52:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-05T22:52:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_12 2023-12-05T22:52:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:52:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-12-05T22:52:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-05T22:52:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_12 deleted 2023-12-05T22:52:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_12 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2023-12-05T22:52:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-12-05T22:52:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-05T22:52:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2023-12-05T22:52:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_13 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-12-05T22:52:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-05T22:52:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_13 deleted 2023-12-05T22:52:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_13 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2023-12-05T22:52:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-12-05T22:52:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-05T22:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2023-12-05T22:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-12-05T22:52:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_14 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-12-05T22:52:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-05T22:52:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_14 deleted 2023-12-05T22:52:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_14 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-12-05T22:52:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-05T22:52:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_15 2023-12-05T22:52:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:52:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-12-05T22:52:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-05T22:52:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_15 deleted 2023-12-05T22:52:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_15 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-12-05T22:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_16 2023-12-05T22:52:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:52:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-12-05T22:52:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-05T22:52:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_16 deleted 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_16 2023-12-05T22:52:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2023-12-05T22:52:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-12-05T22:52:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:52:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_17 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-12-05T22:52:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-05T22:52:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_17 deleted 2023-12-05T22:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_17 2023-12-05T22:52:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2023-12-05T22:52:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-12-05T22:52:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_18 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_18 2023-12-05T22:52:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-05T22:52:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_18 deleted 2023-12-05T22:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_18 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_18 2023-12-05T22:52:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-05T22:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2023-12-05T22:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-12-05T22:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_19 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-12-05T22:52:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-05T22:52:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_19 deleted 2023-12-05T22:52:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_19 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-12-05T22:52:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-05T22:52:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_20 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-12-05T22:52:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-05T22:52:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_20 deleted 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_20 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2023-12-05T22:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-12-05T22:52:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-05T22:52:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_21 2023-12-05T22:52:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:52:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-12-05T22:52:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-05T22:52:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_21 deleted 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_21 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2023-12-05T22:52:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-12-05T22:52:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-05T22:52:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_22 2023-12-05T22:52:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:52:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-12-05T22:52:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-05T22:52:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_22 deleted 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_22 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2023-12-05T22:52:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-12-05T22:52:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_23 2023-12-05T22:52:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:52:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-12-05T22:52:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-05T22:52:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_23 deleted 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_23 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-12-05T22:52:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_24 2023-12-05T22:52:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:52:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-12-05T22:52:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-05T22:52:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_24 deleted 2023-12-05T22:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_24 2023-12-05T22:52:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2023-12-05T22:52:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-12-05T22:52:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_25 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-12-05T22:52:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-05T22:52:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_25 deleted 2023-12-05T22:52:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:52:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_25 2023-12-05T22:52:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2023-12-05T22:52:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-12-05T22:52:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-05T22:52:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_26 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-12-05T22:52:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-05T22:52:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_26 deleted 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_26 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2023-12-05T22:52:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-12-05T22:52:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-05T22:52:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_27 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-12-05T22:52:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-05T22:52:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_27 deleted 2023-12-05T22:52:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:52:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_27 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-12-05T22:52:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-05T22:52:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2023-12-05T22:52:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-12-05T22:52:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_28 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-12-05T22:52:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-05T22:52:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_28 deleted 2023-12-05T22:52:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:52:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_28 2023-12-05T22:52:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2023-12-05T22:52:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-12-05T22:52:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_29 2023-12-05T22:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-12-05T22:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-05T22:52:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_29 deleted 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_29 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2023-12-05T22:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-12-05T22:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-05T22:52:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_30 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:52:50.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-12-05T22:52:50.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-05T22:52:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_30 deleted 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_30 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-12-05T22:52:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_31 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_31 2023-12-05T22:52:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-05T22:52:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_31 deleted 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_31 2023-12-05T22:52:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2023-12-05T22:52:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_31 2023-12-05T22:52:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-05T22:52:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_32 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:52:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-12-05T22:52:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-05T22:52:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_32 deleted 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_32 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2023-12-05T22:52:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-12-05T22:52:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_33 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-12-05T22:52:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-05T22:52:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_33 deleted 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_33 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-12-05T22:52:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-05T22:52:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_34 2023-12-05T22:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:52:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-12-05T22:52:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-05T22:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_34 deleted 2023-12-05T22:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_34 2023-12-05T22:52:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2023-12-05T22:52:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-12-05T22:52:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_35 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-12-05T22:52:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-05T22:52:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_35 deleted 2023-12-05T22:52:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_35 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2023-12-05T22:52:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-12-05T22:52:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-05T22:52:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_36 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:52:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-12-05T22:52:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-05T22:52:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_36 deleted 2023-12-05T22:52:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:52:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_36 2023-12-05T22:52:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2023-12-05T22:52:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-12-05T22:52:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_37 2023-12-05T22:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-12-05T22:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-05T22:52:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_37 deleted 2023-12-05T22:52:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_37 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2023-12-05T22:52:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-12-05T22:52:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:52:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_38 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-12-05T22:52:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-05T22:52:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_38 deleted 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_38 2023-12-05T22:52:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2023-12-05T22:52:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-12-05T22:52:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-05T22:52:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_39 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-12-05T22:52:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-05T22:52:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_39 deleted 2023-12-05T22:52:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_39 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2023-12-05T22:52:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-12-05T22:52:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-05T22:52:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_40 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:52:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-12-05T22:52:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-05T22:52:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_40 deleted 2023-12-05T22:52:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:52:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_40 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-12-05T22:52:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-05T22:52:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_41 2023-12-05T22:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-12-05T22:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-05T22:52:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_41 deleted 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_41 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2023-12-05T22:52:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-12-05T22:52:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_42 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-12-05T22:52:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-05T22:52:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_42 deleted 2023-12-05T22:52:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:52:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_42 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-12-05T22:52:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-05T22:52:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_43 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:52:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-12-05T22:52:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-05T22:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_43 deleted 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_43 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2023-12-05T22:52:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-12-05T22:52:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-05T22:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2023-12-05T22:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-12-05T22:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_44 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-12-05T22:52:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-05T22:52:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_44 deleted 2023-12-05T22:52:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:52:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_44 2023-12-05T22:52:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2023-12-05T22:52:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-12-05T22:52:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:52:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_45 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-12-05T22:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-05T22:52:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_45 deleted 2023-12-05T22:52:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_45 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-12-05T22:52:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_46 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-12-05T22:52:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-05T22:52:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_46 deleted 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_46 2023-12-05T22:52:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2023-12-05T22:52:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-12-05T22:52:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-05T22:52:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_47 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:52:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-12-05T22:52:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-05T22:52:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_47 deleted 2023-12-05T22:52:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:52:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_47 2023-12-05T22:52:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2023-12-05T22:52:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-12-05T22:52:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_48 2023-12-05T22:52:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:52:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-12-05T22:52:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-05T22:52:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_48 deleted 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_48 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2023-12-05T22:52:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-12-05T22:52:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_49 2023-12-05T22:52:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:52:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-12-05T22:52:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-05T22:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_49 deleted 2023-12-05T22:52:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_49 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-12-05T22:52:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:52:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_50 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-12-05T22:52:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-05T22:52:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:52:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:52:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_50 deleted 2023-12-05T22:52:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:52:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:52:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_50 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-12-05T22:52:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:52:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-05T22:52:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:52:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:52:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:52:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:52:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog TEST: delete images during bootstrap 2023-12-05T22:52:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.yK28jjcyo4/rbd-mirror.test.log 2023-12-05T22:52:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ date +%F %T 2023-12-05T22:52:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2023-12-05 22:52:55 TEST: delete images during bootstrap 2023-12-05T22:52:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2023-12-05 22:52:55 TEST: delete images during bootstrap 2023-12-05T22:52:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-12-05T22:52:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=image 2023-12-05T22:52:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-12-05T22:52:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd: mirroring is already configured for image mode 2023-12-05T22:52:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-12-05T22:52:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=image 2023-12-05T22:52:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-12-05T22:52:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd: mirroring is already configured for image mode 2023-12-05T22:52:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1 2023-12-05T22:52:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:52:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1 cluster instance 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=cluster1 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ cluster1 = cluster1 ] 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T22:52:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=test 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 1 10 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_1 512M 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-05T22:52:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_1 2023-12-05T22:52:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-05T22:52:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:52:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_1': 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 18c1e0d139b8 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.18c1e0d139b8 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:52:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:52:55 2023 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:52:55 2023 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:52:55 2023 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b456970f-943f-43ea-8275-d2b20c53362e 2023-12-05T22:52:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:52:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-05T22:52:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:52:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:52:55.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:52:55.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:52:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2023-12-05T22:52:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:52:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:52:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 111226880 bytes 2023-12-05T22:52:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 197083136 bytes 2023-12-05T22:52:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 199034880 bytes 2023-12-05T22:52:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 215187456 bytes 2023-12-05T22:52:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 263588864 bytes 2023-12-05T22:52:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 318033920 bytes 2023-12-05T22:52:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 353045504 bytes 2023-12-05T22:52:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 405499904 bytes 2023-12-05T22:52:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 440952832 bytes 2023-12-05T22:52:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 453272576 bytes 2023-12-05T22:52:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 475021312 bytes 2023-12-05T22:52:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 507752448 bytes 2023-12-05T22:52:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 550738944 bytes 2023-12-05T22:52:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 599170048 bytes 2023-12-05T22:52:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 705646592 bytes 2023-12-05T22:52:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 750526464 bytes 2023-12-05T22:52:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 773123072 bytes 2023-12-05T22:52:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 860462080 bytes 2023-12-05T22:52:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 892846080 bytes 2023-12-05T22:52:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 893452288 bytes 2023-12-05T22:52:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 957314048 bytes 2023-12-05T22:53:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1054776320 bytes 2023-12-05T22:53:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1105549312 bytes 2023-12-05T22:53:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1177433088 bytes 2023-12-05T22:53:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1231434752 bytes 2023-12-05T22:53:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1269772288 bytes 2023-12-05T22:53:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1289291776 bytes 2023-12-05T22:53:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1331801088 bytes 2023-12-05T22:53:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1370893312 bytes 2023-12-05T22:53:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1467337728 bytes 2023-12-05T22:53:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1512940544 bytes 2023-12-05T22:53:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1566023680 bytes 2023-12-05T22:53:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1611567104 bytes 2023-12-05T22:53:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1677900800 bytes 2023-12-05T22:53:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1704394752 bytes 2023-12-05T22:53:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1763299328 bytes 2023-12-05T22:53:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1811241984 bytes 2023-12-05T22:53:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1860898816 bytes 2023-12-05T22:53:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1892892672 bytes 2023-12-05T22:53:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1937363968 bytes 2023-12-05T22:53:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1969425408 bytes 2023-12-05T22:53:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2001889280 bytes 2023-12-05T22:53:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2083163136 bytes 2023-12-05T22:53:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2083765248 bytes 2023-12-05T22:53:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2141869056 bytes 2023-12-05T22:53:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2216826880 bytes 2023-12-05T22:53:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2277049344 bytes 2023-12-05T22:53:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2309365760 bytes 2023-12-05T22:53:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2335651840 bytes 2023-12-05T22:53:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2386813952 bytes 2023-12-05T22:53:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2509493248 bytes 2023-12-05T22:53:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2644201472 bytes 2023-12-05T22:53:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2714158080 bytes 2023-12-05T22:53:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2739320832 bytes 2023-12-05T22:53:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2795180032 bytes 2023-12-05T22:53:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2845380608 bytes 2023-12-05T22:53:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2899050496 bytes 2023-12-05T22:53:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2926125056 bytes 2023-12-05T22:53:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2973123584 bytes 2023-12-05T22:53:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 3013360640 bytes 2023-12-05T22:53:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 3049578496 bytes 2023-12-05T22:53:08.596 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:53:08.592+0000 7fe8efb13640 -1 reset not still connected to 0x55b6187e3c70 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_1 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-05T22:53:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:53:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1289df8b9bf3 2023-12-05T22:53:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:53:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_1 present 1289df8b9bf3 2023-12-05T22:53:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:53:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1289df8b9bf3 2023-12-05T22:53:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:53:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:53:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_1 2023-12-05T22:53:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-05T22:53:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:53:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1289df8b9bf3 2023-12-05T22:53:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1289df8b9bf3 2023-12-05T22:53:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1289df8b9bf3 -o 1289df8b9bf3 = 1289df8b9bf3 2023-12-05T22:53:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:53:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:53:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:53:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_1 2023-12-05T22:53:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-12-05T22:53:29.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:53:29.827+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb95295d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:53:29.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:53:29.827+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb9447b900 [3/b456970f-943f-43ea-8275-d2b20c53362e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:53:40.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:53:40.575+0000 7fdfa10f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026564400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-12-05T22:53:40.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:53:40.575+0000 7fdfa10f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026564400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-12-05T22:53:40.587 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T22:53:40.583+0000 7fe8efb13640 -1 reset not still connected to 0x55b62c9425b0 2023-12-05T22:53:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:53:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-05T22:53:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_1 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-05T22:53:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:53:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-05T22:53:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:53:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:53:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_1 deleted 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_1 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-05T22:53:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:53:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-05T22:53:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:53:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:53:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:53:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:53:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_1 2023-12-05T22:53:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-12-05T22:53:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-05T22:53:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_1 2023-12-05T22:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2023-12-05T22:53:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-05T22:53:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-05T22:53:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_2 512M 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:53:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:53:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:53:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:53:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-05T22:53:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_2 2023-12-05T22:53:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:53:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-05T22:53:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:53:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-05T22:53:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_2': 2023-12-05T22:53:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:53:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:53:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:53:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 18d97aad78d8 2023-12-05T22:53:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.18d97aad78d8 2023-12-05T22:53:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:53:41 2023 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:53:41 2023 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:53:41 2023 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c7c0c83f-4c35-4522-b108-5663ced3c57b 2023-12-05T22:53:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-05T22:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:53:42.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 106520576 bytes 2023-12-05T22:53:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 141102080 bytes 2023-12-05T22:53:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 141711360 bytes 2023-12-05T22:53:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 180448256 bytes 2023-12-05T22:53:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 230362112 bytes 2023-12-05T22:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 282618880 bytes 2023-12-05T22:53:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 305598464 bytes 2023-12-05T22:53:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 324339712 bytes 2023-12-05T22:53:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 363522048 bytes 2023-12-05T22:53:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 370466816 bytes 2023-12-05T22:53:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 396610560 bytes 2023-12-05T22:53:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 412707840 bytes 2023-12-05T22:53:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 471987200 bytes 2023-12-05T22:53:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 513035264 bytes 2023-12-05T22:53:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 545950720 bytes 2023-12-05T22:53:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 639333376 bytes 2023-12-05T22:53:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 678815744 bytes 2023-12-05T22:53:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 709424128 bytes 2023-12-05T22:53:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 710046720 bytes 2023-12-05T22:53:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 747930624 bytes 2023-12-05T22:53:45.602 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5158302201138051) > weight (0.25) 2023-12-05T22:53:45.602 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2575690121846347) > weight (0.25) 2023-12-05T22:53:45.602 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6206657498338389) > weight (0.25) 2023-12-05T22:53:45.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4037025481802756) > weight (0.25) 2023-12-05T22:53:45.603 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-05T22:53:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 791881728 bytes 2023-12-05T22:53:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 917907456 bytes 2023-12-05T22:53:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 949524480 bytes 2023-12-05T22:53:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1006735360 bytes 2023-12-05T22:53:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1086707712 bytes 2023-12-05T22:53:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1087331328 bytes 2023-12-05T22:53:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1109870592 bytes 2023-12-05T22:53:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1148150784 bytes 2023-12-05T22:53:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1175520256 bytes 2023-12-05T22:53:47.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6849200616521259) > weight (0.25) 2023-12-05T22:53:47.406 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-05T22:53:47.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-05T22:53:47.406 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-05T22:53:47.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:53:47.411+0000 7f0529a67640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 61420) UID: 1000 2023-12-05T22:53:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1261523968 bytes 2023-12-05T22:53:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1302676480 bytes 2023-12-05T22:53:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1387725824 bytes 2023-12-05T22:53:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1408866304 bytes 2023-12-05T22:53:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1423009792 bytes 2023-12-05T22:53:48.407 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-05T22:53:48.407 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:53:48.407 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-05T22:53:48.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-05T22:53:48.408 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-05T22:53:48.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-05T22:53:48.410 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-05T22:53:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1461527552 bytes 2023-12-05T22:53:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1507314688 bytes 2023-12-05T22:53:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1527645184 bytes 2023-12-05T22:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1552370688 bytes 2023-12-05T22:53:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1622912000 bytes 2023-12-05T22:53:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1686484992 bytes 2023-12-05T22:53:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1740009472 bytes 2023-12-05T22:53:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1829958656 bytes 2023-12-05T22:53:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1885758464 bytes 2023-12-05T22:53:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1955619840 bytes 2023-12-05T22:53:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1974871040 bytes 2023-12-05T22:53:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1988997120 bytes 2023-12-05T22:53:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2023098368 bytes 2023-12-05T22:53:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2081955840 bytes 2023-12-05T22:53:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2162050048 bytes 2023-12-05T22:53:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2196068352 bytes 2023-12-05T22:53:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2248718336 bytes 2023-12-05T22:53:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2340114432 bytes 2023-12-05T22:53:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2405291008 bytes 2023-12-05T22:53:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2452100096 bytes 2023-12-05T22:53:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2516822016 bytes 2023-12-05T22:53:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2544607232 bytes 2023-12-05T22:53:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2594553856 bytes 2023-12-05T22:53:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2595173376 bytes 2023-12-05T22:53:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2604889088 bytes 2023-12-05T22:53:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2658419712 bytes 2023-12-05T22:53:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2705986560 bytes 2023-12-05T22:53:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2776267776 bytes 2023-12-05T22:53:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2853598208 bytes 2023-12-05T22:53:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2881372160 bytes 2023-12-05T22:53:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2911474688 bytes 2023-12-05T22:53:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2960378880 bytes 2023-12-05T22:53:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3062153216 bytes 2023-12-05T22:53:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3086629888 bytes 2023-12-05T22:53:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3106014208 bytes 2023-12-05T22:53:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3189311488 bytes 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:53:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_2 2023-12-05T22:53:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-05T22:53:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:53:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1452dd7b07d3 2023-12-05T22:53:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_2 present 1452dd7b07d3 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1452dd7b07d3 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:53:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:53:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_2 2023-12-05T22:53:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:53:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-05T22:53:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:53:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1452dd7b07d3 2023-12-05T22:53:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1452dd7b07d3 2023-12-05T22:53:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1452dd7b07d3 -o 1452dd7b07d3 = 1452dd7b07d3 2023-12-05T22:53:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:53:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:53:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:53:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_2 2023-12-05T22:53:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:53:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:53:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:53:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-12-05T22:54:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:54:15.758+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e47b380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:54:15.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:54:15.758+0000 7f86718ea640 -1 rbd::mirror::ImageReplayer: 0x55f03d986000 [3/c7c0c83f-4c35-4522-b108-5663ced3c57b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:54:24.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:54:24.606+0000 7f61f6569640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x561c03912c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-12-05T22:54:24.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:54:24.606+0000 7f61f6569640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x561c03912c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-12-05T22:54:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:54:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:54:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_2 2023-12-05T22:54:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:54:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:54:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: --cluster=cluster1 info mirror/image_2 2023-12-05T22:54:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-05T22:54:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:54:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:54:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_2 2023-12-05T22:54:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:54:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-05T22:54:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:54:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_2 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:54:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-12-05T22:54:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-05T22:54:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:54:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:54:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:54:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_2 2023-12-05T22:54:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:54:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2023-12-05T22:54:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-05T22:54:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-05T22:54:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:54:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_3 512M 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:54:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:54:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-05T22:54:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_3 2023-12-05T22:54:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:54:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:54:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-05T22:54:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:54:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_3': 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 18f5f60f41fc 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.18f5f60f41fc 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:54:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:54:25 2023 2023-12-05T22:54:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:54:25 2023 2023-12-05T22:54:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:54:25 2023 2023-12-05T22:54:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:54:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:54:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 3d78801b-7611-47d9-9718-94b4bd4c80c7 2023-12-05T22:54:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:54:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-05T22:54:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:54:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:54:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:54:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:54:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:54:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 69161984 bytes 2023-12-05T22:54:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 69761024 bytes 2023-12-05T22:54:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 70386688 bytes 2023-12-05T22:54:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 79564800 bytes 2023-12-05T22:54:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 134231040 bytes 2023-12-05T22:54:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 134845440 bytes 2023-12-05T22:54:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 216058880 bytes 2023-12-05T22:54:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 232366080 bytes 2023-12-05T22:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 285658112 bytes 2023-12-05T22:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 340760576 bytes 2023-12-05T22:54:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 383490048 bytes 2023-12-05T22:54:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 418976768 bytes 2023-12-05T22:54:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 439402496 bytes 2023-12-05T22:54:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 484264960 bytes 2023-12-05T22:54:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 548350976 bytes 2023-12-05T22:54:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 578150400 bytes 2023-12-05T22:54:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 638435328 bytes 2023-12-05T22:54:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 661277696 bytes 2023-12-05T22:54:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 701929472 bytes 2023-12-05T22:54:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 744779776 bytes 2023-12-05T22:54:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 818236416 bytes 2023-12-05T22:54:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 839728128 bytes 2023-12-05T22:54:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 890313728 bytes 2023-12-05T22:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 953070592 bytes 2023-12-05T22:54:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1033936896 bytes 2023-12-05T22:54:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1073167360 bytes 2023-12-05T22:54:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1156856832 bytes 2023-12-05T22:54:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1164851200 bytes 2023-12-05T22:54:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1220795392 bytes 2023-12-05T22:54:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1269252096 bytes 2023-12-05T22:54:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1316370432 bytes 2023-12-05T22:54:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1339579392 bytes 2023-12-05T22:54:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1345414144 bytes 2023-12-05T22:54:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1356215296 bytes 2023-12-05T22:54:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1379482624 bytes 2023-12-05T22:54:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1423627264 bytes 2023-12-05T22:54:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1434640384 bytes 2023-12-05T22:54:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1443915776 bytes 2023-12-05T22:54:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1465091072 bytes 2023-12-05T22:54:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1472145408 bytes 2023-12-05T22:54:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1496228864 bytes 2023-12-05T22:54:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1535902720 bytes 2023-12-05T22:54:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1587207168 bytes 2023-12-05T22:54:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1696936960 bytes 2023-12-05T22:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1810096128 bytes 2023-12-05T22:54:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1829593088 bytes 2023-12-05T22:54:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1834417152 bytes 2023-12-05T22:54:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1902718976 bytes 2023-12-05T22:54:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1970978816 bytes 2023-12-05T22:54:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1996078080 bytes 2023-12-05T22:54:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2066027520 bytes 2023-12-05T22:54:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2118606848 bytes 2023-12-05T22:54:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2143576064 bytes 2023-12-05T22:54:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2176557056 bytes 2023-12-05T22:54:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2192536576 bytes 2023-12-05T22:54:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2226811904 bytes 2023-12-05T22:54:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2362559488 bytes 2023-12-05T22:54:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2363179008 bytes 2023-12-05T22:54:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2405275648 bytes 2023-12-05T22:54:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2490406912 bytes 2023-12-05T22:54:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2522311680 bytes 2023-12-05T22:54:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2537144320 bytes 2023-12-05T22:54:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2541976576 bytes 2023-12-05T22:54:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2546194432 bytes 2023-12-05T22:54:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2551266304 bytes 2023-12-05T22:54:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:54:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_3 2023-12-05T22:54:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-05T22:54:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:54:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=12896333b742 2023-12-05T22:54:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:54:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_3 present 12896333b742 2023-12-05T22:54:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=12896333b742 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_3 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-05T22:54:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:54:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=12896333b742 2023-12-05T22:54:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 12896333b742 2023-12-05T22:54:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 12896333b742 -o 12896333b742 = 12896333b742 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_3 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:54:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-12-05T22:54:49.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.37969561478069036) > weight (0.25) 2023-12-05T22:54:49.604 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:54:49.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:54:49.605 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-05T22:54:49.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:54:49.609+0000 7fe34d1f5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 68793) UID: 1000 2023-12-05T22:54:57.606 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:54:57.606 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:54:57.606 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:54:57.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:54:57.606 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:54:57.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:54:57.609 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-05T22:54:59.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:54:59.825+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb955be1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:54:59.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:54:59.825+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb9447b900 [3/3d78801b-7611-47d9-9718-94b4bd4c80c7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:55:07.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:55:07.629+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e0275b7400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-12-05T22:55:07.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:55:07.629+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e0275b7400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-12-05T22:55:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:55:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-05T22:55:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:55:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_3 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + rbd --cluster=cluster1 info mirror/image_3 2023-12-05T22:55:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:55:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-05T22:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:55:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:55:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-12-05T22:55:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:55:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:55:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:55:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:55:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:55:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_3 2023-12-05T22:55:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:55:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-05T22:55:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:55:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_3 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:55:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-12-05T22:55:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-05T22:55:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:55:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:55:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_3 2023-12-05T22:55:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2023-12-05T22:55:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-05T22:55:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-05T22:55:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:55:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:55:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2023-12-05T22:55:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_4 512M 2023-12-05T22:55:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-05T22:55:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_4 2023-12-05T22:55:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:55:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-05T22:55:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:55:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-05T22:55:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_4': 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 191711587caf 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.191711587caf 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:55:08 2023 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:55:08 2023 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:55:08 2023 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:55:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:55:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 3f98f9a7-637f-4775-bde2-4a53f8e65baa 2023-12-05T22:55:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:55:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-05T22:55:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:55:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:55:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:55:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:55:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 42521600 bytes 2023-12-05T22:55:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 79515648 bytes 2023-12-05T22:55:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 87478272 bytes 2023-12-05T22:55:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 143138816 bytes 2023-12-05T22:55:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 179026944 bytes 2023-12-05T22:55:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 235404288 bytes 2023-12-05T22:55:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 260105216 bytes 2023-12-05T22:55:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 329250816 bytes 2023-12-05T22:55:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 373937152 bytes 2023-12-05T22:55:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 419714048 bytes 2023-12-05T22:55:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 475790336 bytes 2023-12-05T22:55:10.997 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T22:55:11.001+0000 7fa63df41640 -1 reset not still connected to 0x5574c7a87450 2023-12-05T22:55:10.997 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T22:55:11.001+0000 7fa63df41640 -1 reset not still connected to 0x5574d970a820 2023-12-05T22:55:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 527225856 bytes 2023-12-05T22:55:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 576304128 bytes 2023-12-05T22:55:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 663599104 bytes 2023-12-05T22:55:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 709732352 bytes 2023-12-05T22:55:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 780944384 bytes 2023-12-05T22:55:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 815532032 bytes 2023-12-05T22:55:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 838204416 bytes 2023-12-05T22:55:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 838828032 bytes 2023-12-05T22:55:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 921972736 bytes 2023-12-05T22:55:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 966945792 bytes 2023-12-05T22:55:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 980780032 bytes 2023-12-05T22:55:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1050188800 bytes 2023-12-05T22:55:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1083256832 bytes 2023-12-05T22:55:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1127502848 bytes 2023-12-05T22:55:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1145972736 bytes 2023-12-05T22:55:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1174295552 bytes 2023-12-05T22:55:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1262756864 bytes 2023-12-05T22:55:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1317275648 bytes 2023-12-05T22:55:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1361199104 bytes 2023-12-05T22:55:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1361822720 bytes 2023-12-05T22:55:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1402894336 bytes 2023-12-05T22:55:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1437613056 bytes 2023-12-05T22:55:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1488440320 bytes 2023-12-05T22:55:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1512995840 bytes 2023-12-05T22:55:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1561039872 bytes 2023-12-05T22:55:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1581959168 bytes 2023-12-05T22:55:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1634408448 bytes 2023-12-05T22:55:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1684616192 bytes 2023-12-05T22:55:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1721093120 bytes 2023-12-05T22:55:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1749511168 bytes 2023-12-05T22:55:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1809214464 bytes 2023-12-05T22:55:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1820486656 bytes 2023-12-05T22:55:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1916055552 bytes 2023-12-05T22:55:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1917521920 bytes 2023-12-05T22:55:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1983197184 bytes 2023-12-05T22:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2010648576 bytes 2023-12-05T22:55:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2091216896 bytes 2023-12-05T22:55:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2117953536 bytes 2023-12-05T22:55:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2141710336 bytes 2023-12-05T22:55:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2156365824 bytes 2023-12-05T22:55:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2185378816 bytes 2023-12-05T22:55:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2205310976 bytes 2023-12-05T22:55:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2239946752 bytes 2023-12-05T22:55:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2245489664 bytes 2023-12-05T22:55:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2249369600 bytes 2023-12-05T22:55:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2298878976 bytes 2023-12-05T22:55:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2327510016 bytes 2023-12-05T22:55:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2361792512 bytes 2023-12-05T22:55:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2409875456 bytes 2023-12-05T22:55:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2439112704 bytes 2023-12-05T22:55:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2474801152 bytes 2023-12-05T22:55:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2514599936 bytes 2023-12-05T22:55:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_4 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-05T22:55:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:55:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1452a6422f23 2023-12-05T22:55:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_4 present 1452a6422f23 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1452a6422f23 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_4 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-05T22:55:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1452a6422f23 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1452a6422f23 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1452a6422f23 -o 1452a6422f23 = 1452a6422f23 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_4 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-12-05T22:55:33.411 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5953569369020473) > weight (0.25) 2023-12-05T22:55:33.411 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6983246042003421) > weight (0.25) 2023-12-05T22:55:33.411 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-05T22:55:33.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-05T22:55:33.412 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-05T22:55:33.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T22:55:33.416+0000 7f799cca4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 66322) UID: 1000 2023-12-05T22:55:35.413 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-05T22:55:35.413 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:55:35.413 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-05T22:55:35.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-05T22:55:35.413 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-05T22:55:35.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-05T22:55:35.416 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-12-05T22:55:45.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:55:45.760+0000 7f866b0dd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e47ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:55:45.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:55:45.760+0000 7f86718ea640 -1 rbd::mirror::ImageReplayer: 0x55f03d986000 [3/3f98f9a7-637f-4775-bde2-4a53f8e65baa] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:55:50.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:55:50.784+0000 7f61f6569640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x561c024ef000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-12-05T22:55:50.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:55:50.784+0000 7f61f6569640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x561c024ef000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-12-05T22:55:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:55:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-05T22:55:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_4 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-05T22:55:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:55:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-05T22:55:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:55:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:55:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_4 deleted 2023-12-05T22:55:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_4 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-05T22:55:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:55:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_4 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-12-05T22:55:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-05T22:55:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_4 2023-12-05T22:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2023-12-05T22:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-05T22:55:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-05T22:55:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:55:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:55:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_5 512M 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:55:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-05T22:55:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_5 2023-12-05T22:55:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:55:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:55:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-05T22:55:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:55:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-05T22:55:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_5': 2023-12-05T22:55:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:55:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:55:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:55:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 193556c9beaa 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.193556c9beaa 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:55:51 2023 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:55:51 2023 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:55:51 2023 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e796bbbf-1ba6-49b0-b795-565d3e6ba197 2023-12-05T22:55:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-05T22:55:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:55:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:55:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:55:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:55:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2023-12-05T22:55:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:55:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:55:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 41777152 bytes 2023-12-05T22:55:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 89379840 bytes 2023-12-05T22:55:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 130552832 bytes 2023-12-05T22:55:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 179457024 bytes 2023-12-05T22:55:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 217062400 bytes 2023-12-05T22:55:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 314307584 bytes 2023-12-05T22:55:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 349373440 bytes 2023-12-05T22:55:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 410192896 bytes 2023-12-05T22:55:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 426533888 bytes 2023-12-05T22:55:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 464408576 bytes 2023-12-05T22:55:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 524303360 bytes 2023-12-05T22:55:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 541398016 bytes 2023-12-05T22:55:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 565548032 bytes 2023-12-05T22:55:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 589779968 bytes 2023-12-05T22:55:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 633222144 bytes 2023-12-05T22:55:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 656617472 bytes 2023-12-05T22:55:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 703817728 bytes 2023-12-05T22:55:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 765883392 bytes 2023-12-05T22:55:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 801644544 bytes 2023-12-05T22:55:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 825771008 bytes 2023-12-05T22:55:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 841224192 bytes 2023-12-05T22:55:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 902417408 bytes 2023-12-05T22:55:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 952098816 bytes 2023-12-05T22:55:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1044466688 bytes 2023-12-05T22:55:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1111723008 bytes 2023-12-05T22:55:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1156140032 bytes 2023-12-05T22:55:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1170439168 bytes 2023-12-05T22:55:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1196279808 bytes 2023-12-05T22:55:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1255240704 bytes 2023-12-05T22:55:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1277979648 bytes 2023-12-05T22:55:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1314422784 bytes 2023-12-05T22:55:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1344361472 bytes 2023-12-05T22:55:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1369632768 bytes 2023-12-05T22:55:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1383421952 bytes 2023-12-05T22:55:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1410475008 bytes 2023-12-05T22:55:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1427930112 bytes 2023-12-05T22:55:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1462595584 bytes 2023-12-05T22:56:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1566956544 bytes 2023-12-05T22:56:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1628592128 bytes 2023-12-05T22:56:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1691062272 bytes 2023-12-05T22:56:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1730570240 bytes 2023-12-05T22:56:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1787737088 bytes 2023-12-05T22:56:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1813873664 bytes 2023-12-05T22:56:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1868374016 bytes 2023-12-05T22:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1907807232 bytes 2023-12-05T22:56:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1924081664 bytes 2023-12-05T22:56:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1953719296 bytes 2023-12-05T22:56:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1974530048 bytes 2023-12-05T22:56:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2004404224 bytes 2023-12-05T22:56:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2028052480 bytes 2023-12-05T22:56:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:56:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:56:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:56:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:56:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-12-05T22:56:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:56:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:56:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:56:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:56:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_5 2023-12-05T22:56:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-05T22:56:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:56:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1289ed0eb85d 2023-12-05T22:56:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_5 present 1289ed0eb85d 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1289ed0eb85d 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_5 2023-12-05T22:56:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-05T22:56:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:56:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1289ed0eb85d 2023-12-05T22:56:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1289ed0eb85d 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1289ed0eb85d -o 1289ed0eb85d = 1289ed0eb85d 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_5 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-12-05T22:56:29.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:56:29.831+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb94a5a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:56:29.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:56:29.831+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447b900 [3/e796bbbf-1ba6-49b0-b795-565d3e6ba197] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:56:33.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:56:33.863+0000 7fdfa10f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026a36000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-12-05T22:56:33.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:56:33.863+0000 7fdfa10f6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026a36000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-12-05T22:56:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:56:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_5 2023-12-05T22:56:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-05T22:56:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:56:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-05T22:56:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:56:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:56:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-12-05T22:56:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:56:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:56:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:56:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:56:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_5 2023-12-05T22:56:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-05T22:56:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_5 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-12-05T22:56:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-05T22:56:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:56:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_5 2023-12-05T22:56:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2023-12-05T22:56:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-05T22:56:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-05T22:56:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_6 512M 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:56:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:56:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:56:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:56:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:56:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-05T22:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_6 2023-12-05T22:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-05T22:56:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:56:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-05T22:56:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_6': 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 194da64868bf 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.194da64868bf 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:56:34 2023 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:56:34 2023 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:56:34 2023 2023-12-05T22:56:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:56:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:56:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 11f5ab66-be8d-44cc-ae27-510c59ff68e8 2023-12-05T22:56:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:56:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-05T22:56:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:56:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:56:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:56:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:56:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 10979328 bytes 2023-12-05T22:56:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 26749952 bytes 2023-12-05T22:56:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 72161280 bytes 2023-12-05T22:56:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 121071616 bytes 2023-12-05T22:56:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 176028672 bytes 2023-12-05T22:56:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 275684352 bytes 2023-12-05T22:56:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 294490112 bytes 2023-12-05T22:56:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 308803584 bytes 2023-12-05T22:56:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 349101056 bytes 2023-12-05T22:56:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 357164032 bytes 2023-12-05T22:56:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 442293248 bytes 2023-12-05T22:56:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 468831232 bytes 2023-12-05T22:56:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 583659520 bytes 2023-12-05T22:56:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 590629888 bytes 2023-12-05T22:56:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 652993536 bytes 2023-12-05T22:56:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 687540224 bytes 2023-12-05T22:56:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 712291328 bytes 2023-12-05T22:56:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 750092288 bytes 2023-12-05T22:56:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 785147904 bytes 2023-12-05T22:56:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 850419712 bytes 2023-12-05T22:56:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 877170688 bytes 2023-12-05T22:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 887323648 bytes 2023-12-05T22:56:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 951498752 bytes 2023-12-05T22:56:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 981137408 bytes 2023-12-05T22:56:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1011972096 bytes 2023-12-05T22:56:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1038049280 bytes 2023-12-05T22:56:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1038672896 bytes 2023-12-05T22:56:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1076162560 bytes 2023-12-05T22:56:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1101249536 bytes 2023-12-05T22:56:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1153819648 bytes 2023-12-05T22:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1203599360 bytes 2023-12-05T22:56:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1267262464 bytes 2023-12-05T22:56:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1325740032 bytes 2023-12-05T22:56:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1386625024 bytes 2023-12-05T22:56:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1420530688 bytes 2023-12-05T22:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1477491712 bytes 2023-12-05T22:56:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1493038080 bytes 2023-12-05T22:56:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1580217344 bytes 2023-12-05T22:56:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1702325248 bytes 2023-12-05T22:56:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1766718464 bytes 2023-12-05T22:56:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1804710912 bytes 2023-12-05T22:56:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1809754112 bytes 2023-12-05T22:56:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1815415808 bytes 2023-12-05T22:56:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1913828352 bytes 2023-12-05T22:56:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1953209344 bytes 2023-12-05T22:56:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1967860736 bytes 2023-12-05T22:56:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2007028736 bytes 2023-12-05T22:56:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2037492736 bytes 2023-12-05T22:56:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2127558656 bytes 2023-12-05T22:56:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2197757952 bytes 2023-12-05T22:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2215793664 bytes 2023-12-05T22:56:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2237536256 bytes 2023-12-05T22:56:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2249133056 bytes 2023-12-05T22:56:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2318345216 bytes 2023-12-05T22:56:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2404241408 bytes 2023-12-05T22:56:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2444002304 bytes 2023-12-05T22:56:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2479751168 bytes 2023-12-05T22:56:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2503841792 bytes 2023-12-05T22:56:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2561853440 bytes 2023-12-05T22:56:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2638971904 bytes 2023-12-05T22:56:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2688108544 bytes 2023-12-05T22:56:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2714138624 bytes 2023-12-05T22:56:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2734248960 bytes 2023-12-05T22:56:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2785889280 bytes 2023-12-05T22:56:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2815764480 bytes 2023-12-05T22:56:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2901787648 bytes 2023-12-05T22:56:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2921091072 bytes 2023-12-05T22:56:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2979361792 bytes 2023-12-05T22:56:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3043757056 bytes 2023-12-05T22:56:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3068539904 bytes 2023-12-05T22:56:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3074743296 bytes 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:56:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:56:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:56:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:56:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:56:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:56:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=14522fe4b5b3 2023-12-05T22:56:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:56:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 present 14522fe4b5b3 2023-12-05T22:56:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:56:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=14522fe4b5b3 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info+ mirror/image_6 2023-12-05T22:56:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:56:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=14522fe4b5b3 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 14522fe4b5b3 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 14522fe4b5b3 -o 14522fe4b5b3 = 14522fe4b5b3 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_6 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:56:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-12-05T22:56:55.610 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9297347134512626) > weight (0.25) 2023-12-05T22:56:55.611 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T22:56:55.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T22:56:55.611 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-05T22:56:55.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T22:56:55.615+0000 7f515fbe9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 73953) UID: 1000 2023-12-05T22:56:57.417 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-05T22:56:57.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-05T22:56:57.418 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-05T22:56:57.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T22:56:57.423+0000 7f6203d84640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 59028) UID: 1000 2023-12-05T22:57:00.612 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T22:57:00.612 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:57:00.613 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T22:57:00.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T22:57:00.613 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T22:57:00.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T22:57:00.616 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-05T22:57:06.418 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-05T22:57:06.418 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:57:06.419 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-05T22:57:06.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-05T22:57:06.419 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-05T22:57:06.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-05T22:57:06.422 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2023-12-05T22:57:15.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:57:15.758+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e2eb520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:57:15.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:57:15.758+0000 7f86720eb640 -1 rbd::mirror::ImageReplayer: 0x55f03d986000 [3/11f5ab66-be8d-44cc-ae27-510c59ff68e8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:57:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:57:18.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:57:18.746+0000 7fdfad90f640 -1 rbd::mirror::ImageMap: 0x55e02607f8c0 handle_peer_ack_remove: failed to remove global_image_id=11f5ab66-be8d-44cc-ae27-510c59ff68e8 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:57:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:57:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:57:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:57:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=14522fe4b5b3 2023-12-05T22:57:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:57:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted 14522fe4b5b3 2023-12-05T22:57:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=14522fe4b5b3 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:57:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=14522fe4b5b3 2023-12-05T22:57:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 14522fe4b5b3 2023-12-05T22:57:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 14522fe4b5b3 -o 14522fe4b5b3 = 14522fe4b5b3 2023-12-05T22:57:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:57:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = present 2023-12-05T22:57:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted 14522fe4b5b3 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=14522fe4b5b3 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:57:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:57:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:57:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=14522fe4b5b3 2023-12-05T22:57:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 14522fe4b5b3 2023-12-05T22:57:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 14522fe4b5b3 -o 14522fe4b5b3 = 14522fe4b5b3 2023-12-05T22:57:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:57:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = present 2023-12-05T22:57:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted 14522fe4b5b3 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=14522fe4b5b3 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:57:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:57:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:57:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=14522fe4b5b3 2023-12-05T22:57:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 14522fe4b5b3 2023-12-05T22:57:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 14522fe4b5b3 -o 14522fe4b5b3 = 14522fe4b5b3 2023-12-05T22:57:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:57:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = present 2023-12-05T22:57:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted 14522fe4b5b3 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=14522fe4b5b3 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:57:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:57:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:57:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=14522fe4b5b3 2023-12-05T22:57:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 14522fe4b5b3 2023-12-05T22:57:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 14522fe4b5b3 -o 14522fe4b5b3 = 14522fe4b5b3 2023-12-05T22:57:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:57:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = present 2023-12-05T22:57:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2023-12-05T22:57:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted 14522fe4b5b3 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=14522fe4b5b3 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_6 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-05T22:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_6 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-12-05T22:57:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-05T22:57:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:57:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:57:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_6 2023-12-05T22:57:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:57:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2023-12-05T22:57:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-05T22:57:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-05T22:57:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_7 512M 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:57:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:57:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:57:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:57:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:57:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-05T22:57:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_7 2023-12-05T22:57:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:57:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:57:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-05T22:57:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:57:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_7': 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1975b0dad46d 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1975b0dad46d 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:57:34 2023 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:57:34 2023 2023-12-05T22:57:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:57:34 2023 2023-12-05T22:57:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:57:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:57:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e9f77d9b-574e-459e-93e2-c59fcd671b19 2023-12-05T22:57:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:57:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-05T22:57:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:57:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:57:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2023-12-05T22:57:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:57:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:57:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 51106816 bytes 2023-12-05T22:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 94087168 bytes 2023-12-05T22:57:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 148211712 bytes 2023-12-05T22:57:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 206238720 bytes 2023-12-05T22:57:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 284512256 bytes 2023-12-05T22:57:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 308855808 bytes 2023-12-05T22:57:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 332198912 bytes 2023-12-05T22:57:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 349353984 bytes 2023-12-05T22:57:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 358855680 bytes 2023-12-05T22:57:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 397603840 bytes 2023-12-05T22:57:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 414627840 bytes 2023-12-05T22:57:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 458382336 bytes 2023-12-05T22:57:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 509574144 bytes 2023-12-05T22:57:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 553288704 bytes 2023-12-05T22:57:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 605326336 bytes 2023-12-05T22:57:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 703141888 bytes 2023-12-05T22:57:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 746397696 bytes 2023-12-05T22:57:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 783681536 bytes 2023-12-05T22:57:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 812691456 bytes 2023-12-05T22:57:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 840620032 bytes 2023-12-05T22:57:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 869423104 bytes 2023-12-05T22:57:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 923245568 bytes 2023-12-05T22:57:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 948575232 bytes 2023-12-05T22:57:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 985371648 bytes 2023-12-05T22:57:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 994023424 bytes 2023-12-05T22:57:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1025582080 bytes 2023-12-05T22:57:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1050226688 bytes 2023-12-05T22:57:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1110479872 bytes 2023-12-05T22:57:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1149620224 bytes 2023-12-05T22:57:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1228367872 bytes 2023-12-05T22:57:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1321460736 bytes 2023-12-05T22:57:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1331422208 bytes 2023-12-05T22:57:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1350697984 bytes 2023-12-05T22:57:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1434068992 bytes 2023-12-05T22:57:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1481004032 bytes 2023-12-05T22:57:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1531622400 bytes 2023-12-05T22:57:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1600658432 bytes 2023-12-05T22:57:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1641282560 bytes 2023-12-05T22:57:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1671674880 bytes 2023-12-05T22:57:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1696129024 bytes 2023-12-05T22:57:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1761453056 bytes 2023-12-05T22:57:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1764257792 bytes 2023-12-05T22:57:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1803143168 bytes 2023-12-05T22:57:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1833308160 bytes 2023-12-05T22:57:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1883254784 bytes 2023-12-05T22:57:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1897679872 bytes 2023-12-05T22:57:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1948432384 bytes 2023-12-05T22:57:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2013928448 bytes 2023-12-05T22:57:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2055780352 bytes 2023-12-05T22:57:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2076560384 bytes 2023-12-05T22:57:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2096891904 bytes 2023-12-05T22:57:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2122634240 bytes 2023-12-05T22:57:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2155433984 bytes 2023-12-05T22:57:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2156048384 bytes 2023-12-05T22:57:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2212597760 bytes 2023-12-05T22:57:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2268142592 bytes 2023-12-05T22:57:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2366852096 bytes 2023-12-05T22:57:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2397191168 bytes 2023-12-05T22:57:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2488110080 bytes 2023-12-05T22:57:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2505595904 bytes 2023-12-05T22:57:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2510752768 bytes 2023-12-05T22:57:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2554296320 bytes 2023-12-05T22:57:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2560400384 bytes 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_7 2023-12-05T22:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-05T22:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=128958cede09 2023-12-05T22:57:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:57:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_7 present 128958cede09 2023-12-05T22:57:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=128958cede09 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_7 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-05T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=128958cede09 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 128958cede09 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 128958cede09 -o 128958cede09 = 128958cede09 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_7 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:57:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-12-05T22:57:59.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:57:59.829+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb94a5a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:57:59.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:57:59.829+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447af00 [3/e9f77d9b-574e-459e-93e2-c59fcd671b19] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:58:14.423 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6465691834140375) > weight (0.25) 2023-12-05T22:58:14.424 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-05T22:58:14.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-05T22:58:14.424 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-05T22:58:14.424 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-05T22:58:14.425 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:58:14.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:58:14.433+0000 7f0c8d3fd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 73511) UID: 1000 2023-12-05T22:58:17.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:58:17.653+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e0275b7400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-12-05T22:58:17.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:58:17.653+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e0275b7400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-12-05T22:58:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:58:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-05T22:58:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:58:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:58:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_7 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:58:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-05T22:58:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-05T22:58:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:58:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_7 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:58:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-05T22:58:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:58:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-05T22:58:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:58:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_7 2023-12-05T22:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-12-05T22:58:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-05T22:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_7 2023-12-05T22:58:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:58:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2023-12-05T22:58:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-05T22:58:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-05T22:58:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2023-12-05T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_8 512M 2023-12-05T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:58:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:58:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:58:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:58:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:58:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:58:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-05T22:58:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_8 2023-12-05T22:58:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:58:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:58:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-05T22:58:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:58:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_8': 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 198d14c4b37d 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.198d14c4b37d 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:58:18 2023 2023-12-05T22:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:58:18 2023 2023-12-05T22:58:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:58:18 2023 2023-12-05T22:58:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:58:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:58:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: efe52c91-8f8e-4fb8-b2a2-742bdf8ec118 2023-12-05T22:58:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:58:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-05T22:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:58:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2023-12-05T22:58:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:58:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:58:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 143802368 bytes 2023-12-05T22:58:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 189968384 bytes 2023-12-05T22:58:20.526 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-05T22:58:20.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-05T22:58:20.527 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-05T22:58:20.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-05T22:58:20.530 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-12-05T22:58:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 223846400 bytes 2023-12-05T22:58:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 251859968 bytes 2023-12-05T22:58:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 293434368 bytes 2023-12-05T22:58:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 308115456 bytes 2023-12-05T22:58:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 344710144 bytes 2023-12-05T22:58:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 377776128 bytes 2023-12-05T22:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 481559552 bytes 2023-12-05T22:58:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 548750336 bytes 2023-12-05T22:58:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 581129216 bytes 2023-12-05T22:58:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 592058368 bytes 2023-12-05T22:58:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 628434944 bytes 2023-12-05T22:58:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 647582720 bytes 2023-12-05T22:58:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 688289792 bytes 2023-12-05T22:58:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 715255808 bytes 2023-12-05T22:58:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 748413952 bytes 2023-12-05T22:58:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 781197312 bytes 2023-12-05T22:58:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 822972416 bytes 2023-12-05T22:58:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 868767744 bytes 2023-12-05T22:58:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 895703040 bytes 2023-12-05T22:58:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 915263488 bytes 2023-12-05T22:58:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 993182720 bytes 2023-12-05T22:58:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1044356096 bytes 2023-12-05T22:58:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1107308544 bytes 2023-12-05T22:58:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1158027264 bytes 2023-12-05T22:58:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1212492800 bytes 2023-12-05T22:58:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1257111552 bytes 2023-12-05T22:58:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1305433088 bytes 2023-12-05T22:58:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1330499584 bytes 2023-12-05T22:58:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1391711232 bytes 2023-12-05T22:58:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1456199680 bytes 2023-12-05T22:58:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1490191360 bytes 2023-12-05T22:58:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1528641536 bytes 2023-12-05T22:58:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1537085440 bytes 2023-12-05T22:58:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1548097536 bytes 2023-12-05T22:58:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1556852736 bytes 2023-12-05T22:58:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1665516544 bytes 2023-12-05T22:58:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1717636096 bytes 2023-12-05T22:58:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1751488512 bytes 2023-12-05T22:58:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1827166208 bytes 2023-12-05T22:58:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1921485824 bytes 2023-12-05T22:58:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1946448896 bytes 2023-12-05T22:58:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1964061696 bytes 2023-12-05T22:58:28.616 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8039980995876254) > weight (0.25) 2023-12-05T22:58:28.617 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5110024612646651) > weight (0.25) 2023-12-05T22:58:28.617 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3510274465075718) > weight (0.25) 2023-12-05T22:58:28.617 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.48491816953725786) > weight (0.25) 2023-12-05T22:58:28.617 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-05T22:58:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1984562176 bytes 2023-12-05T22:58:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:58:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:58:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:58:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:58:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-12-05T22:58:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_8 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-05T22:58:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:58:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=12893646ab2e 2023-12-05T22:58:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:58:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_8 present 12893646ab2e 2023-12-05T22:58:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:58:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=12893646ab2e 2023-12-05T22:58:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:58:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:58:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_8 2023-12-05T22:58:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-05T22:58:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:58:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=12893646ab2e 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 12893646ab2e 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 12893646ab2e -o 12893646ab2e = 12893646ab2e 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_8 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-12-05T22:58:29.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:58:29.833+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb955bfa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:58:29.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:58:29.833+0000 7f5ba6221640 -1 rbd::mirror::ImageReplayer: 0x55cb9447b680 [3/efe52c91-8f8e-4fb8-b2a2-742bdf8ec118] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:58:59.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:58:59.804+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026b85000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-12-05T22:58:59.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T22:58:59.804+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026b85000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-12-05T22:58:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:58:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-05T22:58:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_8 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:58:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:58:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:58:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-05T22:58:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-05T22:58:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:58:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:59:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-12-05T22:59:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_8 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-05T22:59:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:59:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-05T22:59:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:59:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:59:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:59:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_8 2023-12-05T22:59:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:59:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-12-05T22:59:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-05T22:59:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:59:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:59:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_8 2023-12-05T22:59:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2023-12-05T22:59:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-05T22:59:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-05T22:59:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:59:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2023-12-05T22:59:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_9 512M 2023-12-05T22:59:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:59:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:59:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:59:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-05T22:59:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_9 2023-12-05T22:59:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:59:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-05T22:59:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:59:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_9': 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 19a8a838afda 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.19a8a838afda 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:59:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:59:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:59:00 2023 2023-12-05T22:59:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:59:00 2023 2023-12-05T22:59:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:59:00 2023 2023-12-05T22:59:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:59:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:59:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 0300438c-5855-4e74-822e-e7925fbb2141 2023-12-05T22:59:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:59:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-05T22:59:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:59:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2023-12-05T22:59:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:59:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:59:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 30595072 bytes 2023-12-05T22:59:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 64005120 bytes 2023-12-05T22:59:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 120032256 bytes 2023-12-05T22:59:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 201818112 bytes 2023-12-05T22:59:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 293940224 bytes 2023-12-05T22:59:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 370314240 bytes 2023-12-05T22:59:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 399606784 bytes 2023-12-05T22:59:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 464669696 bytes 2023-12-05T22:59:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 537009152 bytes 2023-12-05T22:59:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 537645056 bytes 2023-12-05T22:59:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 538288128 bytes 2023-12-05T22:59:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 572517376 bytes 2023-12-05T22:59:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 628661248 bytes 2023-12-05T22:59:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 640743424 bytes 2023-12-05T22:59:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 671841280 bytes 2023-12-05T22:59:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 720542720 bytes 2023-12-05T22:59:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 789132288 bytes 2023-12-05T22:59:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 840637440 bytes 2023-12-05T22:59:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 868617216 bytes 2023-12-05T22:59:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 908528640 bytes 2023-12-05T22:59:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 969203712 bytes 2023-12-05T22:59:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1007626240 bytes 2023-12-05T22:59:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1072024576 bytes 2023-12-05T22:59:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1130382336 bytes 2023-12-05T22:59:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1133181952 bytes 2023-12-05T22:59:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1178358784 bytes 2023-12-05T22:59:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1205130240 bytes 2023-12-05T22:59:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1277507584 bytes 2023-12-05T22:59:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1338126336 bytes 2023-12-05T22:59:06.762 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:59:06.768+0000 7fdd570d4640 -1 reset not still connected to 0x559d2d5f12b0 2023-12-05T22:59:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1402387456 bytes 2023-12-05T22:59:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1454180352 bytes 2023-12-05T22:59:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1514081280 bytes 2023-12-05T22:59:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1536641024 bytes 2023-12-05T22:59:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1554074624 bytes 2023-12-05T22:59:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1593680896 bytes 2023-12-05T22:59:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1632480256 bytes 2023-12-05T22:59:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1690795008 bytes 2023-12-05T22:59:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1737903104 bytes 2023-12-05T22:59:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1771335680 bytes 2023-12-05T22:59:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1834005504 bytes 2023-12-05T22:59:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1913543680 bytes 2023-12-05T22:59:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1919138816 bytes 2023-12-05T22:59:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1940042752 bytes 2023-12-05T22:59:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1950612480 bytes 2023-12-05T22:59:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2025761792 bytes 2023-12-05T22:59:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2031554560 bytes 2023-12-05T22:59:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2098889728 bytes 2023-12-05T22:59:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2127795200 bytes 2023-12-05T22:59:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2169117696 bytes 2023-12-05T22:59:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_9 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-05T22:59:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:59:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=17d59c83cadb 2023-12-05T22:59:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_9 present 17d59c83cadb 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=17d59c83cadb 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_9 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-05T22:59:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:59:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=17d59c83cadb 2023-12-05T22:59:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 17d59c83cadb 2023-12-05T22:59:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 17d59c83cadb -o 17d59c83cadb = 17d59c83cadb 2023-12-05T22:59:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:59:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:59:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_9 2023-12-05T22:59:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-12-05T22:59:15.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:59:15.760+0000 7f866b8de640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f03e4c7ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:59:15.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T22:59:15.760+0000 7f86718ea640 -1 rbd::mirror::ImageReplayer: 0x55f03e81e780 [3/0300438c-5855-4e74-822e-e7925fbb2141] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T22:59:36.530 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6593175794480555) > weight (0.25) 2023-12-05T22:59:36.531 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-05T22:59:36.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-05T22:59:36.531 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-05T22:59:36.531 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-05T22:59:36.532 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T22:59:36.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T22:59:36.539+0000 7f3aadc26640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 75741) UID: 1000 2023-12-05T22:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T22:59:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-05T22:59:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T22:59:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:59:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:59:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:59:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_9 2023-12-05T22:59:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-05T22:59:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:59:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-05T22:59:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2023-12-05T22:59:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_9 deleted 2023-12-05T22:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_9 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-05T22:59:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:59:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-05T22:59:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_9 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-12-05T22:59:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_9 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2023-12-05T22:59:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-05T22:59:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-05T22:59:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_10 512M 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -n 512M ] 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2023-12-05T22:59:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-05T22:59:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_10 2023-12-05T22:59:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2023-12-05T22:59:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-05T22:59:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2023-12-05T22:59:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-05T22:59:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_10': 2023-12-05T22:59:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2023-12-05T22:59:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2023-12-05T22:59:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2023-12-05T22:59:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 19bfe27bd3f8 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.19bfe27bd3f8 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Tue Dec 5 22:59:41 2023 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Tue Dec 5 22:59:41 2023 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Tue Dec 5 22:59:41 2023 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f88199fd-8aa1-4186-ab3d-7a9f7931866a 2023-12-05T22:59:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2023-12-05T22:59:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-05T22:59:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-05T22:59:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2023-12-05T22:59:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T22:59:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-05T22:59:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 29445120 bytes 2023-12-05T22:59:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 60430336 bytes 2023-12-05T22:59:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 82480128 bytes 2023-12-05T22:59:42.633 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-05T22:59:42.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-05T22:59:42.634 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-05T22:59:42.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-05T22:59:42.636 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-05T22:59:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 119881728 bytes 2023-12-05T22:59:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 184639488 bytes 2023-12-05T22:59:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 205903872 bytes 2023-12-05T22:59:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 261897216 bytes 2023-12-05T22:59:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 287830016 bytes 2023-12-05T22:59:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 333792256 bytes 2023-12-05T22:59:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 387675136 bytes 2023-12-05T22:59:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 435546112 bytes 2023-12-05T22:59:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 474934272 bytes 2023-12-05T22:59:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 537183232 bytes 2023-12-05T22:59:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 539086848 bytes 2023-12-05T22:59:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 561333248 bytes 2023-12-05T22:59:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 571644928 bytes 2023-12-05T22:59:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 594758656 bytes 2023-12-05T22:59:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 600343552 bytes 2023-12-05T22:59:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 669234176 bytes 2023-12-05T22:59:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 707853312 bytes 2023-12-05T22:59:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 721459200 bytes 2023-12-05T22:59:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 754212864 bytes 2023-12-05T22:59:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 779760640 bytes 2023-12-05T22:59:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 798309376 bytes 2023-12-05T22:59:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 820904960 bytes 2023-12-05T22:59:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 860490752 bytes 2023-12-05T22:59:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 903714816 bytes 2023-12-05T22:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 939966464 bytes 2023-12-05T22:59:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 944789504 bytes 2023-12-05T22:59:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 967393280 bytes 2023-12-05T22:59:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1025060864 bytes 2023-12-05T22:59:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1035992064 bytes 2023-12-05T22:59:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1061355520 bytes 2023-12-05T22:59:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1149334528 bytes 2023-12-05T22:59:47.885 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:59:47.891+0000 7fdd5b4b6640 -1 bdev(0x559d221ace00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-05T22:59:47.888 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fdd5b4b6640 time 2023-12-05T22:59:47.895070+0000 2023-12-05T22:59:47.888 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-05T22:59:47.888 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.888 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x559d1fe378c2] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:*** Caught signal (Aborted) ** 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: in thread 7fdd5b4b6640 thread_name:bstore_aio 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:59:47.891+0000 7fdd5b4b6640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fdd5b4b6640 time 2023-12-05T22:59:47.895070+0000 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x559d1fe378c2] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.889 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.890 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.890 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.890 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fdd692b6520] 2023-12-05T22:59:47.890 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: pthread_kill() 2023-12-05T22:59:47.890 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: raise() 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: abort() 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x559d1fe37977] 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 6: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 7: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2023-12-05T22:59:47.895+0000 7fdd5b4b6640 -1 *** Caught signal (Aborted) ** 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: in thread 7fdd5b4b6640 thread_name:bstore_aio 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fdd692b6520] 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: pthread_kill() 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: raise() 2023-12-05T22:59:47.891 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: abort() 2023-12-05T22:59:47.892 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x559d1fe37977] 2023-12-05T22:59:47.892 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 6: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.892 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 7: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.892 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.892 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.892 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-05T22:59:47.892 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.907 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: -396> 2023-12-05T22:59:47.891+0000 7fdd5b4b6640 -1 bdev(0x559d221ace00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: -38> 2023-12-05T22:59:47.891+0000 7fdd5b4b6640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fdd5b4b6640 time 2023-12-05T22:59:47.895070+0000 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x559d1fe378c2] 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.908 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 0> 2023-12-05T22:59:47.895+0000 7fdd5b4b6640 -1 *** Caught signal (Aborted) ** 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: in thread 7fdd5b4b6640 thread_name:bstore_aio 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fdd692b6520] 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: pthread_kill() 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: raise() 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: abort() 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x559d1fe37977] 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 6: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 7: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-05T22:59:47.909 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.917 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: -915> 2023-12-05T22:59:47.891+0000 7fdd5b4b6640 -1 bdev(0x559d221ace00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: -557> 2023-12-05T22:59:47.891+0000 7fdd5b4b6640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fdd5b4b6640 time 2023-12-05T22:59:47.895070+0000 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x559d1fe378c2] 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.918 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: -519> 2023-12-05T22:59:47.895+0000 7fdd5b4b6640 -1 *** Caught signal (Aborted) ** 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: in thread 7fdd5b4b6640 thread_name:bstore_aio 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: ceph version 19.0.0-21-g4a50e96e (4a50e96e7d9d85da28ef008617cb8342574bfd1b) reef (dev) 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fdd692b6520] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2: pthread_kill() 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 3: raise() 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 4: abort() 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x559d1fe37977] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 6: (KernelDevice::_aio_thread()+0x9da) [0x559d2072491a] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 7: ceph-osd(+0x15c41e1) [0x559d207251e1] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdd69308b43] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdd6939aa00] 2023-12-05T22:59:47.919 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-05T22:59:47.920 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr: 2023-12-05T22:59:48.079 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:daemon-helper: command failed with exit status 5 2023-12-05T22:59:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1208750080 bytes 2023-12-05T22:59:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1259449344 bytes 2023-12-05T22:59:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1260058624 bytes 2023-12-05T22:59:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1287369728 bytes 2023-12-05T22:59:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1322603520 bytes 2023-12-05T22:59:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1350150144 bytes 2023-12-05T22:59:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1386574848 bytes 2023-12-05T22:59:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1431055360 bytes 2023-12-05T22:59:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1465584640 bytes 2023-12-05T22:59:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1466188800 bytes 2023-12-05T22:59:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1511587840 bytes 2023-12-05T22:59:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1533304832 bytes 2023-12-05T22:59:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1548217344 bytes 2023-12-05T22:59:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1576875008 bytes 2023-12-05T22:59:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1605239808 bytes 2023-12-05T22:59:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1614344192 bytes 2023-12-05T22:59:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1658290176 bytes 2023-12-05T22:59:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1675836416 bytes 2023-12-05T22:59:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1706438656 bytes 2023-12-05T22:59:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 1739123712 bytes 2023-12-05T22:59:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 1771165696 bytes 2023-12-05T22:59:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 1799137280 bytes 2023-12-05T22:59:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 1844045824 bytes 2023-12-05T22:59:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 1892216832 bytes 2023-12-05T22:59:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 1899963392 bytes 2023-12-05T22:59:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 1912370176 bytes 2023-12-05T22:59:53.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-05T22:59:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 1988258816 bytes 2023-12-05T22:59:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2020260864 bytes 2023-12-05T22:59:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2125681664 bytes 2023-12-05T22:59:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2215345152 bytes 2023-12-05T22:59:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2215954432 bytes 2023-12-05T22:59:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2242569216 bytes 2023-12-05T22:59:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2284487680 bytes 2023-12-05T22:59:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2376740864 bytes 2023-12-05T22:59:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2410859520 bytes 2023-12-05T22:59:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2490714112 bytes 2023-12-05T22:59:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2536383488 bytes 2023-12-05T22:59:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2592835584 bytes 2023-12-05T22:59:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2610395136 bytes 2023-12-05T22:59:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2650133504 bytes 2023-12-05T22:59:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2697200640 bytes 2023-12-05T22:59:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2697823232 bytes 2023-12-05T22:59:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2767298560 bytes 2023-12-05T22:59:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 2784844800 bytes 2023-12-05T22:59:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 2800870400 bytes 2023-12-05T22:59:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2023-12-05T22:59:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T22:59:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 124 -eq 124 ] 2023-12-05T22:59:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_10 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-05T22:59:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:59:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=12896ecbcd2f 2023-12-05T22:59:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_10 present 12896ecbcd2f 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=12896ecbcd2f 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_10 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1+ info mirror/image_10 2023-12-05T22:59:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=12896ecbcd2f 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 12896ecbcd2f 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 12896ecbcd2f -o 12896ecbcd2f = 12896ecbcd2f 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_10 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T22:59:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T22:59:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T22:59:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-12-05T22:59:58.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-05T22:59:59.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:59:59.831+0000 7f5b9f213640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55cb94fe96c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-05T22:59:59.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T22:59:59.831+0000 7f5ba5a20640 -1 rbd::mirror::ImageReplayer: 0x55cb9447ac80 [3/f88199fd-8aa1-4186-ab3d-7a9f7931866a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-05T23:00:03.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-05T23:00:09.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-05T23:00:14.617 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5911211211522746) > weight (0.25) 2023-12-05T23:00:14.618 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-05T23:00:14.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-05T23:00:14.618 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-05T23:00:14.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:14.627+0000 7f92df6f4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 75030) UID: 1000 2023-12-05T23:00:14.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-05T23:00:18.619 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-05T23:00:18.620 INFO:teuthology.orchestra.run:waiting for 600 2023-12-05T23:00:18.620 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-05T23:00:18.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-05T23:00:18.620 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-05T23:00:18.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-05T23:00:18.624 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-05T23:00:20.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-05T23:00:25.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-05T23:00:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:26.830+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026a37800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2023-12-05T23:00:26.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:26.830+0000 7fdfa08f5640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e026a37800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=14fbb4aa-be19-4176-aa5d-6ebec8dd0bb7, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2023-12-05T23:00:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2023-12-05T23:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T23:00:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_10 2023-12-05T23:00:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T23:00:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + rbd --cluster=cluster1sed info -ne mirror/image_10 s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T23:00:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2023-12-05T23:00:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=12896ecbcd2f 2023-12-05T23:00:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2023-12-05T23:00:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_10 deleted 12896ecbcd2f 2023-12-05T23:00:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=12896ecbcd2f 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_image_id cluster1 mirror image_10 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-05T23:00:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-05T23:00:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_10 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T23:00:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2023-12-05T23:00:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-05T23:00:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T23:00:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T23:00:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_10 2023-12-05T23:00:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2023-12-05T23:00:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-05T23:00:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-05T23:00:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-05T23:00:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog TEST: check if removed images' OMAP are removed 2023-12-05T23:00:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.yK28jjcyo4/rbd-mirror.test.log 2023-12-05T23:00:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ date +%F %T 2023-12-05T23:00:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2023-12-05 23:00:27 TEST: check if removed images' OMAP are removed 2023-12-05T23:00:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2023-12-05 23:00:27 TEST: check if removed images' OMAP are removed 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_in_omap cluster1 mirror 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=status_global 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=status_global 2023-12-05T23:00:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-05T23:00:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-05T23:00:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -c status_global 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 0 ] 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_ 2023-12-05T23:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_ 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-05T23:00:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-05T23:00:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -c image_ 2023-12-05T23:00:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2023-12-05T23:00:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T23:00:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 0 ] 2023-12-05T23:00:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2023-12-05T23:00:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local obj_name=rbd_mirror_leader 2023-12-05T23:00:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rados --cluster cluster1 -p mirror listomapkeysgrep rbd_mirror_leader -c 2023-12-05T23:00:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: image_map 2023-12-05T23:00:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 = 0 2023-12-05T23:00:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=1 2023-12-05T23:00:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 -eq 0 ] 2023-12-05T23:00:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2023-12-05T23:00:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-12-05T23:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -c image_map 2023-12-05T23:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 = 0 2023-12-05T23:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=1 2023-12-05T23:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 1 -eq 0 ] 2023-12-05T23:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2023-12-05T23:00:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-05T23:00:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2023-12-05T23:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-05T23:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local pool=mirror 2023-12-05T23:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rados --clustergrep cluster1 -p -c mirror image_map listomapkeys 2023-12-05T23:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: rbd_mirror_leader 2023-12-05T23:00:30.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-05T23:00:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2023-12-05T23:00:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T23:00:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 0 ] 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_in_omap cluster2 mirror 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=status_global 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T23:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=status_global 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-05T23:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -c status_global 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 0 ] 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_ 2023-12-05T23:00:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T23:00:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_ 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + local pool=mirror 2023-12-05T23:00:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2023-12-05T23:00:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -c image_ 2023-12-05T23:00:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-05T23:00:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2023-12-05T23:00:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T23:00:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 0 ] 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2023-12-05T23:00:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ list_omap_keys cluster2 mirror rbd_mirror_leader 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ + rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2023-12-05T23:00:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:grep -c image_map 2023-12-05T23:00:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2023-12-05T23:00:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2023-12-05T23:00:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2023-12-05T23:00:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 0 ] 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cleanup 0 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code=0 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -ne 0 ] 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -z ] 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster instance 2023-12-05T23:00:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-05T23:00:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' does not exist 2023-12-05T23:00:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-05T23:00:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' does not exist 2023-12-05T23:00:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-05T23:00:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' does not exist 2023-12-05T23:00:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-05T23:00:35.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafdf5a7640 -1 rbd::mirror::InstanceWatcher: 0x556008df6700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6101 object: (2) No such file or directory 2023-12-05T23:00:35.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafdfda8640 -1 rbd::mirror::InstanceWatcher: 0x556008e08000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6101 object: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd8d9a640 -1 rbd::mirror::InstanceWatcher: 0x556008df6700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd8d9a640 -1 rbd::mirror::InstanceWatcher: 0x556008e08000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd2d8e640 -1 rbd::mirror::MirrorStatusUpdater 0x556008e6e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd2d8e640 -1 rbd::mirror::MirrorStatusUpdater 0x556008e6e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafdf5a7640 -1 rbd::mirror::NamespaceReplayer: 0x556008df4820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafdf5a7640 -1 rbd::mirror::NamespaceReplayer: 0x556008df49c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd258d640 -1 rbd::mirror::RemotePoolPoller: 0x556007cc5ba0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-05T23:00:35.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd2d8e640 -1 rbd::mirror::RemotePoolPoller: 0x556007cc5ba0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-05T23:00:35.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd2d8e640 -1 rbd::mirror::PoolReplayer: 0x556008a7c000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-05T23:00:35.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd8d9a640 -1 rbd::mirror::MirrorStatusUpdater 0x556008e6e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:35.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafd8d9a640 -1 rbd::mirror::MirrorStatusUpdater 0x556008e6e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:35.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafdfda8640 -1 rbd::mirror::NamespaceReplayer: 0x556008df4820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:35.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:35.698+0000 7fafdfda8640 -1 rbd::mirror::NamespaceReplayer: 0x556008df49c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' does not exist 2023-12-05T23:00:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirrors cluster1 2023-12-05T23:00:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2023-12-05T23:00:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:0 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:1 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2023-12-05T23:00:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:2 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:3 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirrors cluster2 2023-12-05T23:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ seq 0 3 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:0 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:0 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:1 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:1 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:2 2023-12-05T23:00:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:2 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:3 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:3 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ -z 1 ] 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test = /tmp/tmp.yK28jjcyo4 2023-12-05T23:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -Rf /tmp/tmp.yK28jjcyo4 2023-12-05T23:00:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ [ 0 -eq 0 ] 2023-12-05T23:00:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:OK 2023-12-05T23:00:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo OK 2023-12-05T23:00:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ exit 0 2023-12-05T23:00:35.926 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-05T23:00:35.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-12-05T23:00:35.980 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-12-05T23:00:35.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-12-05T23:00:36.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.086+0000 7fafdf5a7640 -1 rbd::mirror::LeaderWatcher: 0x556009867b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:36.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.086+0000 7fafdf5a7640 -1 rbd::mirror::InstanceWatcher: 0x556008a7b880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6101 object: (2) No such file or directory 2023-12-05T23:00:36.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.086+0000 7fafd959b640 -1 rbd::mirror::InstanceWatcher: 0x556008a7b880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.086+0000 7fafd2d8e640 -1 rbd::mirror::MirrorStatusUpdater 0x556008bd45a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.086+0000 7fafdf5a7640 -1 rbd::mirror::NamespaceReplayer: 0x556008df4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.086+0000 7fafd959b640 -1 rbd::mirror::MirrorStatusUpdater 0x556009818c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.086+0000 7fafdfda8640 -1 rbd::mirror::NamespaceReplayer: 0x556008df4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.090+0000 7fafdf5a7640 -1 rbd::mirror::LeaderWatcher: 0x55600919c300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:36.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.090+0000 7fafdf5a7640 -1 rbd::mirror::InstanceWatcher: 0x556008ebbc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6104 object: (2) No such file or directory 2023-12-05T23:00:36.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.090+0000 7fafcb57f640 -1 rbd::mirror::InstanceWatcher: 0x556008ebbc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.114+0000 7fafc5573640 -1 rbd::mirror::RemotePoolPoller: 0x5560090c20d0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-05T23:00:36.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.114+0000 7fafc5573640 -1 rbd::mirror::MirrorStatusUpdater 0x5560090966c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.114+0000 7fafdf5a7640 -1 rbd::mirror::NamespaceReplayer: 0x556009194680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.114+0000 7fafc4d72640 -1 rbd::mirror::RemotePoolPoller: 0x5560090c20d0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-05T23:00:36.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.114+0000 7fafc4d72640 -1 rbd::mirror::PoolReplayer: 0x556008e0e000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-05T23:00:36.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.114+0000 7fafcb57f640 -1 rbd::mirror::MirrorStatusUpdater 0x556009060900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.114+0000 7fafdfda8640 -1 rbd::mirror::NamespaceReplayer: 0x556009194680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-05T23:00:36.330 DEBUG:teuthology.parallel:result is None 2023-12-05T23:00:36.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-05T23:00:36.340 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-05T23:00:36.341 DEBUG:teuthology.orchestra.run.smithi190:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-05T23:00:36.401 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-05T23:00:36.401 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-05T23:00:36.412 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-05T23:00:36.413 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2023-12-05T23:00:36.413 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-05T23:00:36.413 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-05T23:00:36.430 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-05T23:00:36.430 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 20 2023-12-05T23:00:36.430 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-05T23:00:36.430 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:36.440 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-12-05T23:00:36.440 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:36.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5ba6a22640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 56690) UID: 1000 2023-12-05T23:00:36.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b921f9640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cb94d90480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:36.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b9fa14640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cb94d90b40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:36.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b9fa14640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cb947d0fc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:36.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b919f8640 -1 rbd::mirror::PoolWatcher: 0x55cb936fba40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b9f213640 -1 rbd::mirror::PoolWatcher: 0x55cb94c9c000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b9fa14640 -1 rbd::mirror::PoolWatcher: 0x55cb94c9c5a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b8b1eb640 -1 rbd::mirror::PoolWatcher: 0x55cb936fbc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b99207640 -1 rbd::mirror::PoolWatcher: 0x55cb94c9c780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b99207640 -1 rbd::mirror::PoolWatcher: 0x55cb94c9c960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5ba6221640 -1 rbd::mirror::LeaderWatcher: 0x55cb94c78300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-05T23:00:36.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5ba6221640 -1 rbd::mirror::LeaderWatcher: 0x55cb94c78300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-05T23:00:36.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5ba5a20640 -1 rbd::mirror::LeaderWatcher: 0x55cb94c78300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:36.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5ba6221640 -1 rbd::mirror::InstanceWatcher: 0x55cb94900000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5374 object: (2) No such file or directory 2023-12-05T23:00:36.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5ba6221640 -1 rbd::mirror::InstanceWatcher: 0x55cb94896000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5374 object: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5ba5a20640 -1 rbd::mirror::InstanceWatcher: 0x55cb94900700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5377 object: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b9fa14640 -1 rbd::mirror::InstanceWatcher: 0x55cb94900000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b9fa14640 -1 rbd::mirror::InstanceWatcher: 0x55cb94896000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.450+0000 7f5b921f9640 -1 rbd::mirror::InstanceWatcher: 0x55cb94900700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b99207640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb948d0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b99207640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb94b107e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5ba6221640 -1 rbd::mirror::NamespaceReplayer: 0x55cb94876820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5ba6221640 -1 rbd::mirror::NamespaceReplayer: 0x55cb948769c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b8b9ec640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb948d0480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5ba5a20640 -1 rbd::mirror::NamespaceReplayer: 0x55cb94c804e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b9f213640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb948d0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b9f213640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb948d0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5ba5a20640 -1 rbd::mirror::NamespaceReplayer: 0x55cb94876820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5ba5a20640 -1 rbd::mirror::NamespaceReplayer: 0x55cb948769c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b921f9640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb94afe5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5ba6221640 -1 rbd::mirror::NamespaceReplayer: 0x55cb94c804e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b9fa14640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cb94d90d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:36.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.454+0000 7f5b9f213640 -1 rbd::mirror::PoolWatcher: 0x55cb94c9c1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5b99207640 -1 rbd::mirror::PoolWatcher: 0x55cb94c9c3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5ba6221640 -1 rbd::mirror::LeaderWatcher: 0x55cb952ddb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-05T23:00:36.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5ba6221640 -1 rbd::mirror::LeaderWatcher: 0x55cb952ddb00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-05T23:00:36.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5ba5a20640 -1 rbd::mirror::LeaderWatcher: 0x55cb952ddb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:36.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5ba5a20640 -1 rbd::mirror::InstanceWatcher: 0x55cb945fd880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5374 object: (2) No such file or directory 2023-12-05T23:00:36.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5b9f213640 -1 rbd::mirror::InstanceWatcher: 0x55cb945fd880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5b98a06640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb952c86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5ba5a20640 -1 rbd::mirror::NamespaceReplayer: 0x55cb94876000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5b9fa14640 -1 rbd::mirror::MirrorStatusUpdater 0x55cb95264b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.458+0000 7f5ba6221640 -1 rbd::mirror::NamespaceReplayer: 0x55cb94876000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2023-12-05T23:00:36.471 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:36.482 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-12-05T23:00:36.482 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:36.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0fc83640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 65842) UID: 1000 2023-12-05T23:00:36.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0f482640 -1 rbd::mirror::InstanceWatcher: 0x561ddb214380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5851 object: (2) No such file or directory 2023-12-05T23:00:36.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0ec81640 -1 rbd::mirror::InstanceWatcher: 0x561ddb20ae00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5851 object: (2) No such file or directory 2023-12-05T23:00:36.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0f482640 -1 rbd::mirror::InstanceWatcher: 0x561ddb575180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5854 object: (2) No such file or directory 2023-12-05T23:00:36.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd08474640 -1 rbd::mirror::InstanceWatcher: 0x561ddb214380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd08474640 -1 rbd::mirror::InstanceWatcher: 0x561ddb20ae00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fccfb45a640 -1 rbd::mirror::InstanceWatcher: 0x561ddb575180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd01c67640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddb27c7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd01c67640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddb27c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0ec81640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb202820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0ec81640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb2029c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fccf444c640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddb4b27e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0f482640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb6ca000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd08c75640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddb27c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd08c75640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddb27c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0ec81640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb202820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0ec81640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb2029c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fccfac59640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddb482b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.518+0000 7fcd0f482640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb6ca000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.522+0000 7fcd0ec81640 -1 rbd::mirror::LeaderWatcher: 0x561ddbc53b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:36.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.522+0000 7fcd0ec81640 -1 rbd::mirror::InstanceWatcher: 0x561ddaf9d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5851 object: (2) No such file or directory 2023-12-05T23:00:36.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.522+0000 7fcd08474640 -1 rbd::mirror::InstanceWatcher: 0x561ddaf9d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.522+0000 7fcd02468640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddb17e480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.522+0000 7fcd0ec81640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb202000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.522+0000 7fcd08474640 -1 rbd::mirror::MirrorStatusUpdater 0x561ddbc68ea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2023-12-05T23:00:36.522+0000 7fcd0f482640 -1 rbd::mirror::NamespaceReplayer: 0x561ddb202000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2023-12-05T23:00:36.538 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:36.549 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-12-05T23:00:36.550 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:36.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f639f2fe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 76520) UID: 1000 2023-12-05T23:00:36.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f639e2fc640 -1 rbd::mirror::InstanceWatcher: 0x556f5b03d500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6615 object: (2) No such file or directory 2023-12-05T23:00:36.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f639e2fc640 -1 rbd::mirror::InstanceWatcher: 0x556f5b03ce00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6615 object: (2) No such file or directory 2023-12-05T23:00:36.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f6397aef640 -1 rbd::mirror::InstanceWatcher: 0x556f5b03d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f6397aef640 -1 rbd::mirror::InstanceWatcher: 0x556f5b03ce00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f6391ae3640 -1 rbd::mirror::MirrorStatusUpdater 0x556f5b0787e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f6391ae3640 -1 rbd::mirror::MirrorStatusUpdater 0x556f5b078360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f639e2fc640 -1 rbd::mirror::NamespaceReplayer: 0x556f5b01c820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f639e2fc640 -1 rbd::mirror::NamespaceReplayer: 0x556f5b01c9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f6397aef640 -1 rbd::mirror::MirrorStatusUpdater 0x556f5b078000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f6397aef640 -1 rbd::mirror::MirrorStatusUpdater 0x556f5b078120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f639eafd640 -1 rbd::mirror::NamespaceReplayer: 0x556f5b01c820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:36.586+0000 7f639eafd640 -1 rbd::mirror::NamespaceReplayer: 0x556f5b01c9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.654+0000 7fc050f7b640 -1 rbd::mirror::RemotePoolPoller: 0x564bd07e3860 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-05T23:00:36.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.654+0000 7fc05077a640 -1 rbd::mirror::RemotePoolPoller: 0x564bd07e3860 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-05T23:00:36.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.654+0000 7fc05077a640 -1 rbd::mirror::PoolReplayer: 0x564bd16ae000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-05T23:00:36.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05d794640 -1 rbd::mirror::InstanceWatcher: 0x564bd1940e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6139 object: (2) No such file or directory 2023-12-05T23:00:36.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05df95640 -1 rbd::mirror::InstanceWatcher: 0x564bd19a2380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6139 object: (2) No such file or directory 2023-12-05T23:00:36.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc057788640 -1 rbd::mirror::InstanceWatcher: 0x564bd1940e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc056f87640 -1 rbd::mirror::InstanceWatcher: 0x564bd19a2380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05077a640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd1996360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05077a640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd19965a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05df95640 -1 rbd::mirror::NamespaceReplayer: 0x564bd191e820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05df95640 -1 rbd::mirror::NamespaceReplayer: 0x564bd191e9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc056f87640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd1996000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05d794640 -1 rbd::mirror::NamespaceReplayer: 0x564bd191e820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc056f87640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd1996120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.658+0000 7fc05df95640 -1 rbd::mirror::NamespaceReplayer: 0x564bd191e9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.814+0000 7fc05d794640 -1 rbd::mirror::LeaderWatcher: 0x564bd2385b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:36.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.818+0000 7fc05d794640 -1 rbd::mirror::InstanceWatcher: 0x564bd16a9880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6139 object: (2) No such file or directory 2023-12-05T23:00:36.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.818+0000 7fc056f87640 -1 rbd::mirror::InstanceWatcher: 0x564bd16a9880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.822+0000 7fc050f7b640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd23905a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.822+0000 7fc05d794640 -1 rbd::mirror::NamespaceReplayer: 0x564bd191e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.822+0000 7fc056f87640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd2354c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.822+0000 7fc05df95640 -1 rbd::mirror::NamespaceReplayer: 0x564bd191e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.826+0000 7fdfae110640 -1 rbd::mirror::LeaderWatcher: 0x55e0268c1b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-12-05T23:00:36.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.826+0000 7fdfa7903640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e0264f0000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:36.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.826+0000 7fdfa7903640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e026339d40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:36.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.826+0000 7fdfa7903640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e0262446c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:36.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.826+0000 7fc05d794640 -1 rbd::mirror::LeaderWatcher: 0x564bd1d2a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:36.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.826+0000 7fdfa7903640 -1 rbd::mirror::PoolWatcher: 0x55e024ce1860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.826+0000 7fdfa7903640 -1 rbd::mirror::PoolWatcher: 0x55e024ce12c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fdfa7903640 -1 rbd::mirror::PoolWatcher: 0x55e024ce10e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fdfa10f6640 -1 rbd::mirror::PoolWatcher: 0x55e024ce1a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fdfa10f6640 -1 rbd::mirror::PoolWatcher: 0x55e024ce1c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fdfa10f6640 -1 rbd::mirror::PoolWatcher: 0x55e024ce1e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:36.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fc05d794640 -1 rbd::mirror::InstanceWatcher: 0x564bd1a67880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6141 object: (2) No such file or directory 2023-12-05T23:00:36.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fdfad90f640 -1 rbd::mirror::LeaderWatcher: 0x55e0268c1b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-05T23:00:36.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fdfad90f640 -1 rbd::mirror::LeaderWatcher: 0x55e0268c1b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-12-05T23:00:36.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.830+0000 7fc049f6d640 -1 rbd::mirror::InstanceWatcher: 0x564bd1a67880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:36.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.842+0000 7fc043760640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd1b7e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.842+0000 7fc05d794640 -1 rbd::mirror::NamespaceReplayer: 0x564bd1bf6820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.842+0000 7fc043760640 -1 rbd::mirror::RemotePoolPoller: 0x564bd07e3ee0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-12-05T23:00:36.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.842+0000 7fc04976c640 -1 rbd::mirror::MirrorStatusUpdater 0x564bd1b367e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:36.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.842+0000 7fc05df95640 -1 rbd::mirror::NamespaceReplayer: 0x564bd1bf6820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:36.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:36.842+0000 7fc043760640 -1 rbd::mirror::RemotePoolPoller: 0x564bd07e3ee0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-12-05T23:00:38.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:38.358+0000 7fdfae110640 -1 rbd::mirror::LeaderWatcher: 0x55e02623c300 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-12-05T23:00:38.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:38.358+0000 7fdf998e7640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e026a7bb00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-05T23:00:38.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:38.358+0000 7fdf9a0e8640 -1 rbd::mirror::PoolWatcher: 0x55e0265c4960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:38.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:38.358+0000 7fdf930da640 -1 rbd::mirror::PoolWatcher: 0x55e0265c43c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-05T23:00:38.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:38.362+0000 7fdfae110640 -1 rbd::mirror::LeaderWatcher: 0x55e02623c300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-05T23:00:38.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:38.362+0000 7fdfae110640 -1 rbd::mirror::LeaderWatcher: 0x55e02623c300 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-12-05T23:00:39.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.098+0000 7f638aad5640 -1 rbd::mirror::InstanceWatcher: 0x556f5b0af500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:39.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.098+0000 7f6383ac7640 -1 rbd::mirror::MirrorStatusUpdater 0x556f5b078900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:39.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.098+0000 7f639e2fc640 -1 rbd::mirror::NamespaceReplayer: 0x556f5b4124e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:39.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.102+0000 7f639eafd640 -1 rbd::mirror::LeaderWatcher: 0x556f5ba97b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:39.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.106+0000 7f639eafd640 -1 rbd::mirror::InstanceWatcher: 0x556f5adaf880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6615 object: (2) No such file or directory 2023-12-05T23:00:39.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.106+0000 7f63982f0640 -1 rbd::mirror::InstanceWatcher: 0x556f5adaf880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:39.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.106+0000 7f6391ae3640 -1 rbd::mirror::MirrorStatusUpdater 0x556f5ba806c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:39.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.106+0000 7f639eafd640 -1 rbd::mirror::NamespaceReplayer: 0x556f5b01c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:39.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.106+0000 7f6397aef640 -1 rbd::mirror::MirrorStatusUpdater 0x556f5ba22b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:39.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2023-12-05T23:00:39.106+0000 7f639e2fc640 -1 rbd::mirror::NamespaceReplayer: 0x556f5b01c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:41.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-05T23:00:41.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.826+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.826+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.826+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.826+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.830+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.830+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.834+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.834+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.834+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.834+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.838+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.838+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.842+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.842+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.846+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.846+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.850+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.850+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.850+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.850+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.854+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.854+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.854+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.854+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.858+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.858+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.858+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.858+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.862+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.862+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.862+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.862+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.866+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.866+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.866+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.866+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.870+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.870+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.870+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.870+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.874+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.874+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.874+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.874+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.878+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.878+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.878+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.878+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.882+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.882+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.886+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.886+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.886+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.886+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.890+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.890+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.890+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.890+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.894+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.894+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.894+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.894+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.898+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.898+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.906+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.906+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.906+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.906+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.906+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.906+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.910+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.910+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.910+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.910+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.914+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.914+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.914+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.914+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.918+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.918+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.918+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.918+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.922+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.922+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.922+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.922+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.926+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.926+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.926+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.926+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.930+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.930+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.930+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.930+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.934+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.934+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.934+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.934+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.938+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.938+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.942+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.942+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.942+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.942+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.946+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.946+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.946+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.946+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.950+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.950+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.954+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.954+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.954+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.954+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.958+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.958+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.962+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.962+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.962+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.962+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.966+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.966+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.966+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.966+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.970+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.970+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.974+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.974+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.974+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.974+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.978+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.978+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.978+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.978+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.982+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.982+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.982+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.982+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.986+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.986+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.986+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.986+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.990+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.990+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.990+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.990+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.994+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.994+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.994+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.994+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.998+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.998+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.998+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:41.998+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.002+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.002+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:41.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.002+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:41.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.002+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.006+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.006+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.010+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.010+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.010+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.010+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.014+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.014+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.014+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.014+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.018+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.018+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.022+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.022+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.022+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.022+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.026+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.026+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.030+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.030+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.034+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.034+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.034+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.034+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.038+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.038+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.042+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.042+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.042+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.042+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.046+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.046+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.046+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.046+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.050+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.050+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.054+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.054+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.054+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.054+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.058+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.058+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.062+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.062+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.062+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.062+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.066+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.066+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.070+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.070+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.070+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.070+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.074+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.074+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.074+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.078+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.078+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.078+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.082+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.082+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.082+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.082+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.086+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.086+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.090+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.090+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.090+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.090+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.094+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.094+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.094+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.094+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.098+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.098+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.102+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.102+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.102+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.102+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.106+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.106+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.106+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.106+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.110+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.110+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.110+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.110+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.114+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.114+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.118+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.118+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.118+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.118+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.122+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.122+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.122+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.122+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.126+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.126+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.126+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.126+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.130+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.130+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.130+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.130+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.134+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.134+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.134+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.134+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.138+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.138+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.138+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.138+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.142+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.142+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.142+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.142+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.146+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.146+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.146+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.146+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.150+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.150+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.150+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.150+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.154+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.154+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.154+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.154+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.158+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.158+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.158+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.158+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.162+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.162+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.162+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.162+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.166+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.166+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.166+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.166+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.170+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.170+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.174+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.174+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.174+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.174+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.178+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.178+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.178+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.178+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.182+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.182+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.182+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.182+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.182+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.182+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.186+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.186+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.186+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.186+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.186+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.186+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.190+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.190+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.190+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.190+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.190+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.190+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.194+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.198+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.202+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.206+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.210+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.214+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.218+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.222+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.226+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.230+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.234+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.238+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.242+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.246+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.250+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.254+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.258+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.262+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.266+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.270+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.270 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.274+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.278+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.282+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.286+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.290+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.294+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.298+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.302+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.306+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.310+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.314+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.318+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.322+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.326+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.330+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.334+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.338+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.342+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.346+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.350+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.350+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.350+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.350+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.354+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.354+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.354+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.354+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.354+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.354+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.358+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.362+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.366+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.370+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.374+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.378+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.382+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.386+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.390+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.394+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.398+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.402+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.406+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.410+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.414+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.418+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.422+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.426+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.430+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.434+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.438+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.442+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.446+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.450+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.454+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.458+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.462+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.466+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.470+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.474+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.478+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.482+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.486+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.490+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.494+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.498+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.502+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.506+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.510+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.514+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.518+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.522+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.526+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.530+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.534+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.538+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.542+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.546+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.550+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.554+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.558+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.562+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.566+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.570+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.574+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.578+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.582+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.586+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.590+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.594+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.598+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.602+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.606+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.610+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.614+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.618+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.622+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.626+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.630+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.634+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.638+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.642+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.646+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.650+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.654+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-12-05T23:00:42.652 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:42.663 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-12-05T23:00:42.663 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:42.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.658+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.662+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.666+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86728ec640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 64570) UID: 1000 2023-12-05T23:00:42.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03dd5cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:42.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86720eb640 -1 rbd::mirror::InstanceWatcher: 0x55f03dc0e700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5785 object: (2) No such file or directory 2023-12-05T23:00:42.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86720eb640 -1 rbd::mirror::InstanceWatcher: 0x55f03dc0ee00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5785 object: (2) No such file or directory 2023-12-05T23:00:42.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f866b0dd640 -1 rbd::mirror::InstanceWatcher: 0x55f03dc0e700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f866b8de640 -1 rbd::mirror::InstanceWatcher: 0x55f03dc0ee00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f865d8c2640 -1 rbd::mirror::InstanceWatcher: 0x55f03dc0e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86648d0640 -1 rbd::mirror::MirrorStatusUpdater 0x55f03dbe0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86648d0640 -1 rbd::mirror::MirrorStatusUpdater 0x55f03dbe0480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86718ea640 -1 rbd::mirror::NamespaceReplayer: 0x55f03e8d8820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86718ea640 -1 rbd::mirror::NamespaceReplayer: 0x55f03e8d89c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f865e0c3640 -1 rbd::mirror::MirrorStatusUpdater 0x55f03de14a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.670+0000 7f86720eb640 -1 rbd::mirror::NamespaceReplayer: 0x55f03def6680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f866b8de640 -1 rbd::mirror::MirrorStatusUpdater 0x55f03dbe0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f866b8de640 -1 rbd::mirror::MirrorStatusUpdater 0x55f03dbe0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::NamespaceReplayer: 0x55f03e8d8820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::NamespaceReplayer: 0x55f03e8d89c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.674+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f86720eb640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f86718ea640 -1 rbd::mirror::LeaderWatcher: 0x55f03e96c300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:42.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f86718ea640 -1 rbd::mirror::InstanceWatcher: 0x55f03db1b880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5785 object: (2) No such file or directory 2023-12-05T23:00:42.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.678+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f866b8de640 -1 rbd::mirror::InstanceWatcher: 0x55f03db1b880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f86648d0640 -1 rbd::mirror::MirrorStatusUpdater 0x55f03e7f65a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f86718ea640 -1 rbd::mirror::NamespaceReplayer: 0x55f03e8d8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f866b8de640 -1 rbd::mirror::MirrorStatusUpdater 0x55f03e798c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f86720eb640 -1 rbd::mirror::NamespaceReplayer: 0x55f03e8d8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.682+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.686+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.686+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.686+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.686+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.690+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.694+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.698+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.698+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.698+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.698+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.698+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.698+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.702+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.706+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.710+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.710+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.710+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.710+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2023-12-05T23:00:42.707 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:42.718 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-12-05T23:00:42.718 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:42.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.710+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.710+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.714+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.718+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.722+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.722+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.722+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.722+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.722+0000 7fdfae911640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 48220) UID: 1000 2023-12-05T23:00:42.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.722+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.722+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfad90f640 -1 rbd::mirror::LeaderWatcher: 0x55e02623c300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:42.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfae110640 -1 rbd::mirror::InstanceWatcher: 0x55e025e4c700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4745 object: (2) No such file or directory 2023-12-05T23:00:42.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfae110640 -1 rbd::mirror::InstanceWatcher: 0x55e02625a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4745 object: (2) No such file or directory 2023-12-05T23:00:42.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.726+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.726+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfae110640 -1 rbd::mirror::InstanceWatcher: 0x55e025e4d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4748 object: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfa7102640 -1 rbd::mirror::InstanceWatcher: 0x55e025e4c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfa7903640 -1 rbd::mirror::InstanceWatcher: 0x55e02625a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfa08f5640 -1 rbd::mirror::MirrorStatusUpdater 0x55e02607e900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfa08f5640 -1 rbd::mirror::MirrorStatusUpdater 0x55e02607eb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfad90f640 -1 rbd::mirror::NamespaceReplayer: 0x55e0262409c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfad90f640 -1 rbd::mirror::NamespaceReplayer: 0x55e026240b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdf9a0e8640 -1 rbd::mirror::InstanceWatcher: 0x55e025e4d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.726+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.726+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfa7903640 -1 rbd::mirror::MirrorStatusUpdater 0x55e02607e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfa7903640 -1 rbd::mirror::MirrorStatusUpdater 0x55e02607e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfad90f640 -1 rbd::mirror::NamespaceReplayer: 0x55e0262409c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.726+0000 7fdfad90f640 -1 rbd::mirror::NamespaceReplayer: 0x55e026240b60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.730+0000 7fdf930da640 -1 rbd::mirror::MirrorStatusUpdater 0x55e02607f440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.730+0000 7fdfae110640 -1 rbd::mirror::NamespaceReplayer: 0x55e0262404e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.730+0000 7fdf998e7640 -1 rbd::mirror::MirrorStatusUpdater 0x55e0260399e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.730+0000 7fdfad90f640 -1 rbd::mirror::NamespaceReplayer: 0x55e0262404e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.730+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.730+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.730+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.730+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.734+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.734+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.734+0000 7fdfae110640 -1 rbd::mirror::LeaderWatcher: 0x55e0268c1b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:42.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.734+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.734+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.734+0000 7fdfae110640 -1 rbd::mirror::InstanceWatcher: 0x55e025bc5880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4745 object: (2) No such file or directory 2023-12-05T23:00:42.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.734+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.734+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.734+0000 7fdfa7102640 -1 rbd::mirror::InstanceWatcher: 0x55e025bc5880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.738+0000 7fdfa08f5640 -1 rbd::mirror::MirrorStatusUpdater 0x55e0268a66c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.738+0000 7fdfae110640 -1 rbd::mirror::NamespaceReplayer: 0x55e025e34000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.738+0000 7fdfa7102640 -1 rbd::mirror::MirrorStatusUpdater 0x55e02682ca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2023-12-05T23:00:42.738+0000 7fdfad90f640 -1 rbd::mirror::NamespaceReplayer: 0x55e025e34000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.738+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.738+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.738+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.738+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.738+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.738+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.742+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.742+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.742+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.742+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.746+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.746+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.746+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.746+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.746+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.746+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.750+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.750+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.750+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.750+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.750+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.750+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.754+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.758+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.762+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.766+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.766+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.766+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.766+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.766+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.766+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.770+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.770+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2023-12-05T23:00:42.765 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:42.777 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-12-05T23:00:42.777 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:42.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.774+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.778+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.778+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.778+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.778+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.778+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.778+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.782+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.782+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2023-12-05T23:00:42.782+0000 7fafe05a9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 74424) UID: 1000 2023-12-05T23:00:42.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.782+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.782+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.782+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.782+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.786+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.790+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.790+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.790+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.790+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.790+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.790+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.794+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.794+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.794+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.794+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.798+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.798+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.798+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.798+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.802+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.802+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.802+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.802+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.802+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.802+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2023-12-05T23:00:42.799 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:42.810 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-12-05T23:00:42.810 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:42.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.806+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.810+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.810+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.810+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.810+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.814+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.814+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.814+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.814+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.814+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.814+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.814+0000 7f9a5dd42640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 76362) UID: 1000 2023-12-05T23:00:42.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::LeaderWatcher: 0x562341efa300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:42.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::InstanceWatcher: 0x562341aed180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6211 object: (2) No such file or directory 2023-12-05T23:00:42.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::InstanceWatcher: 0x562341aca000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6211 object: (2) No such file or directory 2023-12-05T23:00:42.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a56d34640 -1 rbd::mirror::InstanceWatcher: 0x562341aed180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a56d34640 -1 rbd::mirror::InstanceWatcher: 0x562341aca000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::InstanceWatcher: 0x562341acb500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6214 object: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a50527640 -1 rbd::mirror::MirrorStatusUpdater 0x562341b46360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a50527640 -1 rbd::mirror::MirrorStatusUpdater 0x562341b465a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5d541640 -1 rbd::mirror::NamespaceReplayer: 0x562341ac8b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5d541640 -1 rbd::mirror::NamespaceReplayer: 0x562341ac89c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a56533640 -1 rbd::mirror::MirrorStatusUpdater 0x562341b46120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a56533640 -1 rbd::mirror::MirrorStatusUpdater 0x562341b46000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::NamespaceReplayer: 0x562341ac8b60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::NamespaceReplayer: 0x562341ac89c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a49519640 -1 rbd::mirror::InstanceWatcher: 0x562341acb500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a4250b640 -1 rbd::mirror::MirrorStatusUpdater 0x562341d5e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::NamespaceReplayer: 0x562341efc4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a48d18640 -1 rbd::mirror::MirrorStatusUpdater 0x562341cfc6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.818+0000 7f9a5cd40640 -1 rbd::mirror::NamespaceReplayer: 0x562341efc4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.822+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.822+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.822+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-12-05T23:00:42.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.822+0000 7f9a5d541640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-12-05T23:00:42.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.826+0000 7f9a5cd40640 -1 rbd::mirror::LeaderWatcher: 0x562342533b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-05T23:00:42.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.826+0000 7f9a5cd40640 -1 rbd::mirror::InstanceWatcher: 0x562341857880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6211 object: (2) No such file or directory 2023-12-05T23:00:42.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.826+0000 7f9a56d34640 -1 rbd::mirror::InstanceWatcher: 0x562341857880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-05T23:00:42.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.826+0000 7f9a4fd26640 -1 rbd::mirror::MirrorStatusUpdater 0x5623424f07e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.826+0000 7f9a5cd40640 -1 rbd::mirror::NamespaceReplayer: 0x562341ac8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.826+0000 7f9a56d34640 -1 rbd::mirror::MirrorStatusUpdater 0x5623424aaa20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-05T23:00:42.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2023-12-05T23:00:42.826+0000 7f9a5d541640 -1 rbd::mirror::NamespaceReplayer: 0x562341ac8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-05T23:00:42.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2023-12-05T23:00:42.850 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-05T23:00:42.902 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-12-05T23:00:42.902 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:00:42.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2023-12-05T23:00:42.906+0000 7fc05e796640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 75121) UID: 1000 2023-12-05T23:00:42.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2023-12-05T23:00:42.926 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-05T23:00:42.941 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-05T23:00:42.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:43.272 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:43.272 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:43.290 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1709,"stamp":"2023-12-05T23:00:41.632712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547456,"kb_used_data":303000,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568064,"statfs":{"total":289910292480,"available":286277697536,"internally_reserved":0,"allocated":310272000,"data_stored":300455769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,2,0,1,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":471,"apply_latency_ms":471,"commit_latency_ns":471000000,"apply_latency_ns":471000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2252,"num_read_kb":-2135,"num_write":-436,"num_write_kb":-337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.823665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":520,"reported_epoch":506,"state":"active+clean","last_fresh":"2023-12-05T23:00:26.827892+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:26.827892+0000","last_peered":"2023-12-05T23:00:26.827892+0000","last_clean":"2023-12-05T23:00:26.827892+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:26.827892+0000","last_undegraded":"2023-12-05T23:00:26.827892+0000","last_fullsized":"2023-12-05T23:00:26.827892+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":506,"state":"active+clean","last_fresh":"2023-12-05T23:00:26.825899+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:26.825899+0000","last_peered":"2023-12-05T23:00:26.825899+0000","last_clean":"2023-12-05T23:00:26.825899+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:26.825899+0000","last_undegraded":"2023-12-05T23:00:26.825899+0000","last_fullsized":"2023-12-05T23:00:26.825899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":656,"reported_epoch":507,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:33.252153+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:33.252153+0000","last_peered":"2023-12-05T23:00:33.252153+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:33.252153+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937928,"kb_used_data":5888,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433912,"statfs":{"total":96636764160,"available":95676325888,"internally_reserved":0,"allocated":6029312,"data_stored":3010511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,2,0,1,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":233,"apply_latency_ms":233,"commit_latency_ns":233000000,"apply_latency_ns":233000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.436}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042892,"kb_used_data":6468,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328948,"statfs":{"total":96636764160,"available":95568842752,"internally_reserved":0,"allocated":6623232,"data_stored":3600879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":232,"apply_latency_ms":232,"commit_latency_ns":232000000,"apply_latency_ns":232000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:43.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:43.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:43.581 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:43.599 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1709,"stamp":"2023-12-05T23:00:41.632712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547456,"kb_used_data":303000,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568064,"statfs":{"total":289910292480,"available":286277697536,"internally_reserved":0,"allocated":310272000,"data_stored":300455769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,2,0,1,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":471,"apply_latency_ms":471,"commit_latency_ns":471000000,"apply_latency_ns":471000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2252,"num_read_kb":-2135,"num_write":-436,"num_write_kb":-337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.823665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":520,"reported_epoch":506,"state":"active+clean","last_fresh":"2023-12-05T23:00:26.827892+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:26.827892+0000","last_peered":"2023-12-05T23:00:26.827892+0000","last_clean":"2023-12-05T23:00:26.827892+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:26.827892+0000","last_undegraded":"2023-12-05T23:00:26.827892+0000","last_fullsized":"2023-12-05T23:00:26.827892+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":506,"state":"active+clean","last_fresh":"2023-12-05T23:00:26.825899+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:26.825899+0000","last_peered":"2023-12-05T23:00:26.825899+0000","last_clean":"2023-12-05T23:00:26.825899+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:26.825899+0000","last_undegraded":"2023-12-05T23:00:26.825899+0000","last_fullsized":"2023-12-05T23:00:26.825899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":656,"reported_epoch":507,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:33.252153+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:33.252153+0000","last_peered":"2023-12-05T23:00:33.252153+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:33.252153+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937928,"kb_used_data":5888,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433912,"statfs":{"total":96636764160,"available":95676325888,"internally_reserved":0,"allocated":6029312,"data_stored":3010511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,2,0,1,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":233,"apply_latency_ms":233,"commit_latency_ns":233000000,"apply_latency_ns":233000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.436}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042892,"kb_used_data":6468,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328948,"statfs":{"total":96636764160,"available":95568842752,"internally_reserved":0,"allocated":6623232,"data_stored":3600879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":232,"apply_latency_ms":232,"commit_latency_ns":232000000,"apply_latency_ns":232000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:43.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:43.887 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:43.887 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:43.906 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1710,"stamp":"2023-12-05T23:00:43.633261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":160,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547472,"kb_used_data":303012,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568048,"statfs":{"total":289910292480,"available":286277681152,"internally_reserved":0,"allocated":310284288,"data_stored":300462418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":243,"apply_latency_ms":243,"commit_latency_ns":243000000,"apply_latency_ns":243000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2252,"num_read_kb":-2135,"num_write":-436,"num_write_kb":-337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002869"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":656,"reported_epoch":507,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:33.252153+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:33.252153+0000","last_peered":"2023-12-05T23:00:33.252153+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:33.252153+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.52400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042892,"kb_used_data":6468,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328948,"statfs":{"total":96636764160,"available":95568842752,"internally_reserved":0,"allocated":6623232,"data_stored":3600879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":232,"apply_latency_ms":232,"commit_latency_ns":232000000,"apply_latency_ns":232000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:43.907 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:00:43.907 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:43.907 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:00:43.907 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:43.907 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:00:43.907 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:43.908 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:00:43.908 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:43.908 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:00:43.908 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:43.908 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:00:43.908 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 656, 'reported_epoch': 507, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:33.252153+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:33.252153+0000', 'last_peered': '2023-12-05T23:00:33.252153+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:33.252153+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:43.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:00:44.239 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:44.239 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2419,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632611328,"bytes_avail":286277681152,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:00:44.256 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:00:44.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:44.543 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:44.544 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:44.562 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1710,"stamp":"2023-12-05T23:00:43.633261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":160,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547472,"kb_used_data":303012,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568048,"statfs":{"total":289910292480,"available":286277681152,"internally_reserved":0,"allocated":310284288,"data_stored":300462418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":243,"apply_latency_ms":243,"commit_latency_ns":243000000,"apply_latency_ns":243000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2252,"num_read_kb":-2135,"num_write":-436,"num_write_kb":-337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002869"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":656,"reported_epoch":507,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:33.252153+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:33.252153+0000","last_peered":"2023-12-05T23:00:33.252153+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:33.252153+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.52400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042892,"kb_used_data":6468,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328948,"statfs":{"total":96636764160,"available":95568842752,"internally_reserved":0,"allocated":6623232,"data_stored":3600879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":232,"apply_latency_ms":232,"commit_latency_ns":232000000,"apply_latency_ns":232000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:47.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-05T23:00:47.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:47.857 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:47.858 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:47.874 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1712,"stamp":"2023-12-05T23:00:47.634294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.817624"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":656,"reported_epoch":507,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:33.252153+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:33.252153+0000","last_peered":"2023-12-05T23:00:33.252153+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:33.252153+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.66000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:47.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:48.164 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:48.165 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:48.182 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1712,"stamp":"2023-12-05T23:00:47.634294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.817624"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":656,"reported_epoch":507,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:33.252153+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:33.252153+0000","last_peered":"2023-12-05T23:00:33.252153+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:33.252153+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.66000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:48.183 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:00:48.183 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:48.184 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:00:48.185 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 656, 'reported_epoch': 507, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:33.252153+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:33.252153+0000', 'last_peered': '2023-12-05T23:00:33.252153+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:33.252153+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:48.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:00:48.510 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:48.510 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2423,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:00:48.527 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:00:48.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:48.789 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:48.789 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:48.805 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1712,"stamp":"2023-12-05T23:00:47.634294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.817624"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":656,"reported_epoch":507,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:33.252153+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:33.252153+0000","last_peered":"2023-12-05T23:00:33.252153+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:33.252153+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.66000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:51.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:52.109 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:52.109 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:52.127 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1714,"stamp":"2023-12-05T23:00:51.635366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.66000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.629}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:52.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:52.412 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:52.412 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:52.429 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1714,"stamp":"2023-12-05T23:00:51.635366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.66000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.629}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:52.429 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:00:52.430 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:52.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:00:52.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-05T23:00:52.759 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:52.759 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2427,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:00:52.778 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:00:52.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:53.256 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:53.256 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:53.273 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1714,"stamp":"2023-12-05T23:00:51.635366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.66000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.629}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:56.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:56.566 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:56.566 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:56.583 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1716,"stamp":"2023-12-05T23:00:55.636568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003306"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:56.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:56.858 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:56.874 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1716,"stamp":"2023-12-05T23:00:55.636568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003306"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:56.875 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:00:56.875 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:56.875 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:00:56.875 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:56.875 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:00:56.875 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:56.875 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:00:56.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:56.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:00:56.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:56.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:00:56.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:00:56.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:00:57.196 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:57.196 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2432,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:00:57.215 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:00:57.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:00:57.495 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:00:57.496 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:00:57.512 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1716,"stamp":"2023-12-05T23:00:55.636568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003306"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.826338+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:32.826338+0000","last_peered":"2023-12-05T23:00:32.826338+0000","last_clean":"2023-12-05T23:00:32.826338+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:32.826338+0000","last_undegraded":"2023-12-05T23:00:32.826338+0000","last_fullsized":"2023-12-05T23:00:32.826338+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.42599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:00:57.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-05T23:01:00.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:00.809 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:00.809 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:00.827 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1718,"stamp":"2023-12-05T23:00:59.637811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.503},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:00.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:01.100 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:01.101 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:01.117 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1718,"stamp":"2023-12-05T23:00:59.637811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.503},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:01.117 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:01.117 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:01.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:01.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:01.447 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:01.448 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2436,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:01.465 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:01.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:01.728 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:01.728 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:01.745 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1719,"stamp":"2023-12-05T23:01:01.638427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.503},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:03.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-05T23:01:04.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:05.048 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:05.049 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:05.067 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1720,"stamp":"2023-12-05T23:01:03.639007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003640"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:05.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:05.348 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:05.348 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:05.365 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1720,"stamp":"2023-12-05T23:01:03.639007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003640"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:05.365 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:05.366 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:05.367 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:05.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:05.698 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:05.698 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2440,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:05.715 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:05.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:05.990 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:05.990 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:06.007 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1721,"stamp":"2023-12-05T23:01:05.639594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.67300000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:08.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-05T23:01:09.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:09.304 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:09.305 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:09.323 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1722,"stamp":"2023-12-05T23:01:07.640220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003652"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.46899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.67300000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:09.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:09.605 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:09.606 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:09.622 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1722,"stamp":"2023-12-05T23:01:07.640220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003652"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.46899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.67300000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:09.623 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:09.623 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:09.623 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:09.623 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:09.623 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:09.623 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:09.623 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:09.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:09.624 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:09.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:09.624 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:09.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:09.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:09.947 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:09.947 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2444,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:09.966 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:09.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:10.226 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:10.226 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:10.243 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1723,"stamp":"2023-12-05T23:01:09.640824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003621"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.46899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:13.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:13.536 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:13.536 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:13.552 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1724,"stamp":"2023-12-05T23:01:11.641446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003635"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.173,"5min":3.9340000000000002,"15min":5.0140000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.032,"15min":0.78200000000000003},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.5129999999999999,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.46899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:13.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:13.842 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:13.860 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1725,"stamp":"2023-12-05T23:01:13.642012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003585"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.499},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.56499999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:13.861 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:13.861 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:13.861 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:13.862 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:13.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:14.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-05T23:01:14.207 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:14.207 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2449,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:14.226 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:14.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:14.511 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:14.512 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:14.530 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1725,"stamp":"2023-12-05T23:01:13.642012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003585"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.499},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.56499999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.81999999999999995,"15min":0.70799999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.71199999999999997,"15min":0.65300000000000002},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:17.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:17.830 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:17.831 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:17.850 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1727,"stamp":"2023-12-05T23:01:17.643194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738860,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:17.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:18.141 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:18.141 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:18.157 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1727,"stamp":"2023-12-05T23:01:17.643194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738860,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:18.158 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:18.158 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:18.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:18.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:18.476 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:18.476 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2453,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:18.495 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:18.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:18.767 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:18.768 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:18.785 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1727,"stamp":"2023-12-05T23:01:17.643194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738860,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:19.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~87s 2023-12-05T23:01:21.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:22.086 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:22.087 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:22.104 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1729,"stamp":"2023-12-05T23:01:21.644416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:22.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:22.385 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:22.403 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1729,"stamp":"2023-12-05T23:01:21.644416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:22.403 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:22.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:22.405 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:22.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:22.739 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2457,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:22.759 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:22.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:23.041 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:23.041 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:23.057 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1729,"stamp":"2023-12-05T23:01:21.644416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:25.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-05T23:01:26.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:26.358 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:26.359 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:26.377 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1731,"stamp":"2023-12-05T23:01:25.645614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003601"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.56299999999999994}]}]},{"osd":1,"up_from":8,"seq":34359738862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:26.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:26.661 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:26.678 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1731,"stamp":"2023-12-05T23:01:25.645614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003601"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.56299999999999994}]}]},{"osd":1,"up_from":8,"seq":34359738862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:26.678 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:26.679 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:26.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:27.005 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2461,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:27.022 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:27.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:27.283 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:27.283 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:27.299 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1731,"stamp":"2023-12-05T23:01:25.645614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003601"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.56299999999999994}]}]},{"osd":1,"up_from":8,"seq":34359738862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:30.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:30.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-05T23:01:30.601 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:30.601 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:30.619 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1733,"stamp":"2023-12-05T23:01:29.646820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003625"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:30.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:30.906 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:30.907 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:30.923 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1733,"stamp":"2023-12-05T23:01:29.646820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003625"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:30.924 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:30.924 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:30.924 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:30.924 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:30.925 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:30.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:31.245 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:31.246 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2466,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:31.261 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:31.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:31.531 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:31.531 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:31.548 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1733,"stamp":"2023-12-05T23:01:29.646820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003625"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:34.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:34.847 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:34.848 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:34.866 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1735,"stamp":"2023-12-05T23:01:33.648011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003595"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738864,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:34.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:35.152 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:35.152 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:35.169 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1735,"stamp":"2023-12-05T23:01:33.648011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003595"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738864,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:35.170 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:35.171 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:35.171 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:35.171 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:35.171 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:35.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:35.490 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:35.490 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2470,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:35.507 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:35.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:35.784 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:35.785 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:35.801 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1736,"stamp":"2023-12-05T23:01:35.648623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003612"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738864,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.44700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.498},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56599999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:35.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-05T23:01:38.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:39.090 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:39.091 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:39.107 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1737,"stamp":"2023-12-05T23:01:37.649365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.47499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.498},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56599999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:39.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:39.386 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:39.403 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1737,"stamp":"2023-12-05T23:01:37.649365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.47499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.498},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56599999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:39.403 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:39.404 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:39.405 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:39.405 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:39.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:39.714 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2474,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:39.732 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:39.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:40.014 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:40.015 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:40.031 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1738,"stamp":"2023-12-05T23:01:39.650114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003866"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.47499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:41.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-05T23:01:43.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:43.325 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:43.325 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:43.341 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1739,"stamp":"2023-12-05T23:01:41.650889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004069"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.47499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:43.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:43.619 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:43.619 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:43.636 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1739,"stamp":"2023-12-05T23:01:41.650889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004069"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.47499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:43.637 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:43.637 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:43.637 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:43.637 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:43.637 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:43.637 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:43.637 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:43.638 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:43.638 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:43.638 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:43.638 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:43.638 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:43.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:43.963 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:43.963 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2478,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:43.981 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:43.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:44.260 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:44.260 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:44.277 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1740,"stamp":"2023-12-05T23:01:43.651629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004218"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.439},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:46.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~114s 2023-12-05T23:01:47.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:47.578 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:47.578 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:47.598 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1741,"stamp":"2023-12-05T23:01:45.652372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004361"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.439},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:47.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:47.872 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:47.872 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:47.889 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1742,"stamp":"2023-12-05T23:01:47.653124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004501"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.245695+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:33.245695+0000', 'last_peered': '2023-12-05T23:00:33.245695+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:33.245695+0000', 'last_undegraded': '2023-12-05T23:00:33.245695+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:47.890 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:47.891 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:47.891 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:47.891 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:47.891 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:47.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:48.228 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:48.229 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2483,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:48.248 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:48.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:48.510 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:48.510 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:48.527 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1742,"stamp":"2023-12-05T23:01:47.653124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004501"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.245695+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:33.245695+0000","last_peered":"2023-12-05T23:00:33.245695+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:33.245695+0000","last_undegraded":"2023-12-05T23:00:33.245695+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:51.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:51.817 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:51.831 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1744,"stamp":"2023-12-05T23:01:51.654407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004293"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.318},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:51.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:52.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-05T23:01:52.126 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:52.126 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:52.143 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1744,"stamp":"2023-12-05T23:01:51.654407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004293"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.318},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:52.144 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 525, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.244178+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:33.244178+0000', 'last_peered': '2023-12-05T23:00:33.244178+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:33.244178+0000', 'last_undegraded': '2023-12-05T23:00:33.244178+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.251897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:33.251897+0000', 'last_peered': '2023-12-05T23:00:33.251897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:33.251897+0000', 'last_undegraded': '2023-12-05T23:00:33.251897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:52.145 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:52.146 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:52.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:52.473 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:52.474 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2487,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:52.491 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:52.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:52.773 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:52.774 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:52.790 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1744,"stamp":"2023-12-05T23:01:51.654407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004293"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":525,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.244178+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:33.244178+0000","last_peered":"2023-12-05T23:00:33.244178+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:33.244178+0000","last_undegraded":"2023-12-05T23:00:33.244178+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.251897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:33.251897+0000","last_peered":"2023-12-05T23:00:33.251897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:33.251897+0000","last_undegraded":"2023-12-05T23:00:33.251897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.318},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:55.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:56.093 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:56.111 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1746,"stamp":"2023-12-05T23:01:55.655624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003994"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.502}]}]},{"osd":1,"up_from":8,"seq":34359738868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.441},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:56.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:56.403 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:56.404 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:56.422 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1746,"stamp":"2023-12-05T23:01:55.655624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003994"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.502}]}]},{"osd":1,"up_from":8,"seq":34359738868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.441},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:56.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:01:56.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:56.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:01:56.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:56.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:01:56.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:56.425 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:01:56.425 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:56.425 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:01:56.425 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:56.425 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:01:56.425 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:01:56.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:01:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:56.768 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2491,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:01:56.785 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:01:56.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:01:57.055 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:01:57.056 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:01:57.073 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1746,"stamp":"2023-12-05T23:01:55.655624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003994"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.502}]}]},{"osd":1,"up_from":8,"seq":34359738868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.441},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:01:57.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-05T23:02:00.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:00.383 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:00.384 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:00.402 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1748,"stamp":"2023-12-05T23:01:59.656893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.69099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:00.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:00.682 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:00.683 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:00.699 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1748,"stamp":"2023-12-05T23:01:59.656893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.69099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:00.701 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:00.701 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:00.701 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:00.702 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:00.702 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:00.702 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:00.702 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:00.702 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:00.702 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:00.703 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:00.703 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:00.703 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:00.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:01.046 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:01.047 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2495,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:01.065 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:01.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:01.328 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:01.328 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:01.345 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1748,"stamp":"2023-12-05T23:01:59.656893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.69099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:02.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-05T23:02:04.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:04.627 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:04.627 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:04.643 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1750,"stamp":"2023-12-05T23:02:03.658175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738870,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.498}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.69099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:04.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:04.903 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:04.903 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:04.920 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1750,"stamp":"2023-12-05T23:02:03.658175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738870,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.498}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.69099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:04.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:04.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:04.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:04.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:04.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:04.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:04.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:04.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:04.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:04.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:04.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:04.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:04.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:05.249 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:05.250 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2500,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:05.267 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:05.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:05.533 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:05.534 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:05.550 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1750,"stamp":"2023-12-05T23:02:03.658175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738870,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.498}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.69099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:08.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-05T23:02:08.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:08.839 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:08.840 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:08.856 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1752,"stamp":"2023-12-05T23:02:07.659430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003805"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.313},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.498},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.57299999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:08.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:09.140 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:09.140 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:09.158 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1752,"stamp":"2023-12-05T23:02:07.659430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003805"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.313},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.498},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.57299999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:09.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:09.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:09.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:09.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:09.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:09.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:09.487 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:09.487 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2504,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:09.505 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:09.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:09.781 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:09.782 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:09.799 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1753,"stamp":"2023-12-05T23:02:09.660036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.313},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.54500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:12.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:13.090 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:13.091 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:13.108 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1754,"stamp":"2023-12-05T23:02:11.660602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003709"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.313},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.54500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:13.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:13.382 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:13.398 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1754,"stamp":"2023-12-05T23:02:11.660602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003709"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:01:08 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.67500000000000004,"15min":0.60799999999999998},"min":{"1min":0.28899999999999998,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":11.629,"5min":11.629,"15min":11.629},"last":0.313},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.103,"15min":0.80200000000000005},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":11.593,"5min":21.992000000000001,"15min":21.992000000000001},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.54500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:13.400 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:13.400 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:13.400 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:13.400 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:13.400 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:13.401 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:13.401 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:13.401 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:13.401 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:13.401 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:13.401 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:13.401 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:13.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:13.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2023-12-05T23:02:13.728 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:13.729 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2508,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:13.746 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:13.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:14.034 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:14.034 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:14.049 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1755,"stamp":"2023-12-05T23:02:13.661236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003706"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.75800000000000001,"15min":0.70399999999999996},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.61499999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.64700000000000002},"min":{"1min":0.24299999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.68200000000000005,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.54500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:17.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:17.342 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:17.343 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:17.359 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1756,"stamp":"2023-12-05T23:02:15.661858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003683"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738872,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:17.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:17.621 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:17.621 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:17.639 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1756,"stamp":"2023-12-05T23:02:15.661858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003683"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:32.823870+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:32.823870+0000","last_peered":"2023-12-05T23:00:32.823870+0000","last_clean":"2023-12-05T23:00:32.823870+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:32.823870+0000","last_undegraded":"2023-12-05T23:00:32.823870+0000","last_fullsized":"2023-12-05T23:00:32.823870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738872,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:17.639 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:17.640 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:17.641 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:17.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:17.978 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:17.979 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2512,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:17.995 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:17.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:18.279 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:18.279 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:18.297 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1757,"stamp":"2023-12-05T23:02:17.662485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58999999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738872,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:19.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-05T23:02:21.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:21.595 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:21.613 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1758,"stamp":"2023-12-05T23:02:19.663078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003648"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58999999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:21.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:21.895 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:21.895 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1759,"stamp":"2023-12-05T23:02:21.663732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003696"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58999999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:21.912 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:21.912 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:21.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:21.914 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:21.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:22.250 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:22.250 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2517,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:22.267 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:22.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:22.542 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:22.543 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:22.559 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1759,"stamp":"2023-12-05T23:02:21.663732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003696"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":522,"reported_epoch":507,"state":"active+clean","last_fresh":"2023-12-05T23:00:33.251758+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:33.251758+0000","last_peered":"2023-12-05T23:00:33.251758+0000","last_clean":"2023-12-05T23:00:33.251758+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:33.251758+0000","last_undegraded":"2023-12-05T23:00:33.251758+0000","last_fullsized":"2023-12-05T23:00:33.251758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58999999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:24.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-05T23:02:25.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:25.844 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:25.844 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:25.860 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1761,"stamp":"2023-12-05T23:02:25.664954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:25.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:26.146 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:26.165 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1761,"stamp":"2023-12-05T23:02:25.664954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:26.166 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:26.166 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:26.166 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:26.167 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:26.167 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:26.167 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:26.167 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:26.167 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.238687+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:33.238687+0000', 'last_peered': '2023-12-05T23:00:33.238687+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:33.238687+0000', 'last_undegraded': '2023-12-05T23:00:33.238687+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:26.168 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:26.168 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:26.168 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:26.168 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:26.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:26.499 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:26.499 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2521,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:26.516 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:26.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:26.793 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:26.794 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:26.811 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1761,"stamp":"2023-12-05T23:02:25.664954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.238687+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:33.238687+0000","last_peered":"2023-12-05T23:00:33.238687+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:33.238687+0000","last_undegraded":"2023-12-05T23:00:33.238687+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:29.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:29.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-05T23:02:30.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:30.112 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:30.129 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1763,"stamp":"2023-12-05T23:02:29.666255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.434},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.504}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:30.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:30.406 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:30.406 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:30.422 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1763,"stamp":"2023-12-05T23:02:29.666255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.434},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.504}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:30.423 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:30.424 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:30.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:30.737 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:30.738 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2525,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:30.755 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:30.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:31.028 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:31.028 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:31.044 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1763,"stamp":"2023-12-05T23:02:29.666255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.434},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.504}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:34.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:34.346 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:34.347 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:34.365 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1765,"stamp":"2023-12-05T23:02:33.667509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003776"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.51800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:34.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:34.650 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:34.650 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:34.668 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1765,"stamp":"2023-12-05T23:02:33.667509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003776"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.51800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:34.669 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:34.669 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 507, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:33.243348+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:33.243348+0000', 'last_peered': '2023-12-05T23:00:33.243348+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:33.243348+0000', 'last_undegraded': '2023-12-05T23:00:33.243348+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:34.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:34.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:34.988 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2529,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:35.006 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:35.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:35.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-05T23:02:35.291 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:35.291 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:35.309 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1765,"stamp":"2023-12-05T23:02:33.667509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003776"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":526,"reported_epoch":507,"state":"active+undersized","last_fresh":"2023-12-05T23:00:33.243348+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:33.243348+0000","last_peered":"2023-12-05T23:00:33.243348+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:33.243348+0000","last_undegraded":"2023-12-05T23:00:33.243348+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.51800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:38.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:38.607 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:38.623 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1767,"stamp":"2023-12-05T23:02:37.668736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003782"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738876,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:38.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:38.947 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T23:02:38.956+0000 7fd8cbfff640 0 --2- 172.21.15.190:0/2201310773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8cc148b20 0x7fd8cc06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-05T23:02:39.127 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:39.128 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:39.146 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1767,"stamp":"2023-12-05T23:02:37.668736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003782"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738876,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:39.147 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:39.148 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:39.148 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:39.148 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:39.148 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:39.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:39.514 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2534,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:39.515 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:39.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:39.801 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:39.818 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1768,"stamp":"2023-12-05T23:02:39.669359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003759"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.435}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:40.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-05T23:02:42.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:43.122 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:43.123 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:43.141 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1769,"stamp":"2023-12-05T23:02:41.670031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.435}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:43.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:43.426 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:43.426 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:43.444 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1769,"stamp":"2023-12-05T23:02:41.670031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.58199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.435}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:43.445 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:43.445 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:43.445 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:43.445 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:43.445 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:43.445 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:43.445 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:43.446 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:43.446 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:43.446 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:43.446 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:43.446 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:43.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:43.767 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:43.767 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2538,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:43.785 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:43.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:44.050 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:44.050 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:44.067 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1770,"stamp":"2023-12-05T23:02:43.670663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738878,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.435}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:46.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-05T23:02:47.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:47.367 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:47.368 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:47.384 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2023-12-05T23:02:45.671399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003890"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738878,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738878,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.53300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:47.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:47.658 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:47.659 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:47.675 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2023-12-05T23:02:45.671399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003890"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738878,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738878,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.53300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:47.676 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:47.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:47.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:47.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:47.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 657, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:39.386696+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:39.386696+0000', 'last_peered': '2023-12-05T23:00:39.386696+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:39.386696+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:47.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:48.020 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2542,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:48.039 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:48.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:48.314 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:48.314 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:48.330 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1772,"stamp":"2023-12-05T23:02:47.672022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003904"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":657,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:39.386696+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:39.386696+0000","last_peered":"2023-12-05T23:00:39.386696+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:39.386696+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738878,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.53300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:51.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:51.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-05T23:02:51.675 INFO:teuthology.orchestra.run.smithi190.stderr:2023-12-05T23:02:51.684+0000 7f99137fe640 0 --2- 172.21.15.190:0/3839931710 >> v2:172.21.15.190:3300/0 conn(0x7f9914148b20 0x7f99140a9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-05T23:02:51.838 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:51.838 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:51.855 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1774,"stamp":"2023-12-05T23:02:51.673522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004162"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.29899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:51.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:52.140 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:52.140 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:52.158 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1774,"stamp":"2023-12-05T23:02:51.673522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004162"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.29899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:52.159 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:52.159 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:52.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:52.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:52.485 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:52.485 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2547,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:52.503 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:52.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:52.773 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:52.773 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:52.789 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1774,"stamp":"2023-12-05T23:02:51.673522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004162"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.29899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:55.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:56.080 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:56.097 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1776,"stamp":"2023-12-05T23:02:55.675035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004371"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.49399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738880,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:56.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:56.366 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:56.366 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:56.382 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1776,"stamp":"2023-12-05T23:02:55.675035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004371"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.49399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738880,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:02:56.383 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:02:56.383 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:56.383 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:02:56.383 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:56.383 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:02:56.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:56.384 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:02:56.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:56.384 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:02:56.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:56.384 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:02:56.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:02:56.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:02:56.699 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:56.700 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2551,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:02:56.717 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:02:56.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:02:56.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-05T23:02:56.994 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:02:56.994 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:02:57.010 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1776,"stamp":"2023-12-05T23:02:55.675035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004371"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.363150+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:37.363150+0000","last_peered":"2023-12-05T23:00:37.363150+0000","last_clean":"2023-12-05T23:00:37.363150+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:37.363150+0000","last_undegraded":"2023-12-05T23:00:37.363150+0000","last_fullsized":"2023-12-05T23:00:37.363150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.49399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738880,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:00.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:00.304 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:00.305 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:00.321 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1778,"stamp":"2023-12-05T23:02:59.676503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004481"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:00.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:00.609 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:00.609 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:00.627 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1778,"stamp":"2023-12-05T23:02:59.676503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004481"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:00.628 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:00.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:00.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:00.950 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:00.950 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2555,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:00.967 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:00.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:01.249 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:01.250 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:01.267 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1778,"stamp":"2023-12-05T23:02:59.676503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004481"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:02.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-05T23:03:04.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:04.553 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:04.554 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:04.570 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1780,"stamp":"2023-12-05T23:03:03.677979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004457"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738882,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.61199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:04.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:04.856 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:04.857 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:04.874 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1780,"stamp":"2023-12-05T23:03:03.677979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004457"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738882,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.61199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:04.875 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:04.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:04.877 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:04.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:05.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:05.204 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2560,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:05.221 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:05.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:05.503 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:05.520 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1780,"stamp":"2023-12-05T23:03:03.677979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004457"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738882,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.61199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:07.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-05T23:03:08.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:08.823 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:08.823 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:08.842 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1782,"stamp":"2023-12-05T23:03:07.679286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004251"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.626}]}]},{"osd":1,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:08.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:09.114 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:09.130 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1782,"stamp":"2023-12-05T23:03:07.679286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004251"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.626}]}]},{"osd":1,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:09.131 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:09.131 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:09.132 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:09.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:09.448 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:09.448 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2564,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:09.465 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:09.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:09.732 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:09.733 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:09.748 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1783,"stamp":"2023-12-05T23:03:09.680031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004265"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.626}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:12.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:13.046 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:13.046 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:13.064 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2023-12-05T23:03:11.680664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004160"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.626}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:13.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:13.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-05T23:03:13.344 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:13.345 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:13.363 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2023-12-05T23:03:11.680664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004160"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.66500000000000004,"15min":0.59799999999999998},"min":{"1min":0.251,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.61599999999999999,"5min":11.629,"15min":11.629},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.85299999999999998,"15min":0.78700000000000003},"min":{"1min":0.31,"5min":0.22,"15min":0.22},"max":{"1min":0.57999999999999996,"5min":12.166,"15min":21.992000000000001},"last":0.626}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:13.364 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:13.365 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:13.365 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:13.365 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:13.365 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:13.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:13.675 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:13.675 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2568,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:13.693 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:13.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:13.961 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:13.978 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1785,"stamp":"2023-12-05T23:03:13.681407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004157"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738884,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:16.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:17.284 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:17.285 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:17.303 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1786,"stamp":"2023-12-05T23:03:15.682169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004190"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738884,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.432},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:17.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:17.585 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:17.585 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:17.604 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1786,"stamp":"2023-12-05T23:03:15.682169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004190"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738884,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.432},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:17.604 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:17.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:17.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:17.926 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:17.927 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2572,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:17.944 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:17.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:18.231 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:18.231 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:18.249 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1787,"stamp":"2023-12-05T23:03:17.682909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004311"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.73699999999999999,"15min":0.69999999999999996},"min":{"1min":0.318,"5min":0.186,"15min":0.186},"max":{"1min":0.61699999999999999,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.432},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.65500000000000003,"15min":0.64000000000000001},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.215},"max":{"1min":0.69099999999999995,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:18.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-05T23:03:21.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:21.541 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:21.541 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:21.557 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1788,"stamp":"2023-12-05T23:03:19.683499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004212"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.36599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:21.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:21.829 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:21.846 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1789,"stamp":"2023-12-05T23:03:21.684073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004041"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.36599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:21.847 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:21.847 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:21.848 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:21.848 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:21.848 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:21.848 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:21.849 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:21.849 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:21.849 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:21.849 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:21.849 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:21.849 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:21.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:22.169 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2576,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:22.187 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:22.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:22.472 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:22.473 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:22.491 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1789,"stamp":"2023-12-05T23:03:21.684073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004041"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.36599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:23.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-05T23:03:25.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:25.798 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:25.799 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:25.817 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1791,"stamp":"2023-12-05T23:03:25.685239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003832"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.42399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738886,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:25.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:26.085 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:26.101 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1791,"stamp":"2023-12-05T23:03:25.685239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003832"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.42399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738886,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:26.102 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:26.102 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:26.102 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:26.102 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:26.102 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:26.103 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:26.103 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:26.103 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:26.103 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:26.103 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:26.103 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:26.103 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:26.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:26.415 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:26.416 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2581,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:26.433 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:26.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:26.702 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:26.718 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1791,"stamp":"2023-12-05T23:03:25.685239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003832"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.42399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738886,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:29.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-05T23:03:29.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:30.021 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:30.040 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1793,"stamp":"2023-12-05T23:03:29.686493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:30.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:30.325 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:30.326 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:30.342 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1793,"stamp":"2023-12-05T23:03:29.686493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:30.343 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:30.343 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:30.343 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:30.343 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:30.343 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:30.343 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:30.344 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:30.344 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:30.344 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:30.344 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:30.344 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:30.344 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:30.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:30.666 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:30.706 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2585,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:30.708 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:30.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:30.978 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:30.979 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:30.995 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1793,"stamp":"2023-12-05T23:03:29.686493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:33.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:34.293 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:34.294 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:34.312 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1795,"stamp":"2023-12-05T23:03:33.687665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.51900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:34.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:34.606 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:34.606 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:34.625 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1795,"stamp":"2023-12-05T23:03:33.687665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.51900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:34.626 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:34.626 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:34.626 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:34.627 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:34.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:34.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-12-05T23:03:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:34.958 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2589,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:34.975 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:34.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:35.237 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:35.238 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:35.254 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1795,"stamp":"2023-12-05T23:03:33.687665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.51900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:38.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:38.549 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:38.566 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1797,"stamp":"2023-12-05T23:03:37.688890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003651"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.65000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:38.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:38.857 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:38.875 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1797,"stamp":"2023-12-05T23:03:37.688890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003651"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.65000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:38.876 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:38.877 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:38.877 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:38.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2594,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:39.213 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:39.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:39.494 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1797,"stamp":"2023-12-05T23:03:37.688890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003651"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.65000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.437},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:40.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-05T23:03:42.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:42.810 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:42.810 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:42.827 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1799,"stamp":"2023-12-05T23:03:41.690093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.65000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.36199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:42.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:43.088 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:43.088 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:43.105 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1799,"stamp":"2023-12-05T23:03:41.690093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.65000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.36199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:43.106 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:43.107 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:43.107 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:43.107 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:43.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:43.447 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:43.447 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2598,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:43.466 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:43.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:43.745 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:43.745 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:43.761 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1800,"stamp":"2023-12-05T23:03:43.690704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003636"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.498},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.57999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.36199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:45.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-05T23:03:46.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:47.058 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:47.058 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:47.077 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1801,"stamp":"2023-12-05T23:03:45.691315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003649"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.498},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.57999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:47.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:47.357 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:47.357 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:47.375 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1801,"stamp":"2023-12-05T23:03:45.691315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003649"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.498},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.57999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:47.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:47.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:47.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:47.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:47.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:47.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:47.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:47.377 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:47.377 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:47.377 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:47.377 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:47.377 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:47.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:47.704 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:47.704 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2602,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:47.721 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:47.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:47.993 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:48.009 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1802,"stamp":"2023-12-05T23:03:47.691935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003651"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.51800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:50.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-12-05T23:03:51.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:51.300 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:51.301 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:51.317 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1803,"stamp":"2023-12-05T23:03:49.692525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003635"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.435},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:51.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:51.606 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:51.607 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:51.625 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1803,"stamp":"2023-12-05T23:03:49.692525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003635"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.435},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:51.626 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:51.627 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:51.627 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:51.627 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:51.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:51.966 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:51.966 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2606,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:51.985 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:51.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:52.249 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:52.249 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:52.266 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1804,"stamp":"2023-12-05T23:03:51.693123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003622"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.435},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:55.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:55.554 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:55.570 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1805,"stamp":"2023-12-05T23:03:53.693711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003618"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.435},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:55.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:55.833 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:55.833 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:55.850 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1806,"stamp":"2023-12-05T23:03:55.694292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738892,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.57699999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:55.850 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.850038+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:00:38.850038+0000', 'last_peered': '2023-12-05T23:00:38.850038+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:00:38.850038+0000', 'last_undegraded': '2023-12-05T23:00:38.850038+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:39.400897+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:00:39.400897+0000', 'last_peered': '2023-12-05T23:00:39.400897+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:00:39.400897+0000', 'last_undegraded': '2023-12-05T23:00:39.400897+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:03:55.851 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:03:55.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:03:56.196 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:56.196 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2611,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:03:56.215 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:03:56.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:56.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-05T23:03:56.486 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:56.487 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:56.504 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1806,"stamp":"2023-12-05T23:03:55.694292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.850038+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:00:38.850038+0000","last_peered":"2023-12-05T23:00:38.850038+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:00:38.850038+0000","last_undegraded":"2023-12-05T23:00:38.850038+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:39.400897+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:00:39.400897+0000","last_peered":"2023-12-05T23:00:39.400897+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:00:39.400897+0000","last_undegraded":"2023-12-05T23:00:39.400897+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.55200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738892,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.57699999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:59.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:03:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:03:59.804 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:03:59.823 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1808,"stamp":"2023-12-05T23:03:59.695497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003562"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738893,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:03:59.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:00.100 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:00.101 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:00.117 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1808,"stamp":"2023-12-05T23:03:59.695497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003562"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738893,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:00.117 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:00.117 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:00.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:00.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:00.467 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:00.468 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2615,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:00.487 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:00.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:00.754 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:00.755 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:00.771 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1808,"stamp":"2023-12-05T23:03:59.695497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003562"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738893,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:01.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-12-05T23:04:03.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:04.070 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:04.070 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:04.089 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1810,"stamp":"2023-12-05T23:04:03.696717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003594"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.68300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738893,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:04.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:04.376 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:04.377 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:04.395 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1810,"stamp":"2023-12-05T23:04:03.696717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003594"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.68300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738893,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 526, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.981305+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:00:38.981305+0000', 'last_peered': '2023-12-05T23:00:38.981305+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:00:38.981305+0000', 'last_undegraded': '2023-12-05T23:00:38.981305+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:04.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:04.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:04.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:04.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:04.733 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:04.733 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2619,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:04.750 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:04.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:05.026 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:05.027 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:05.043 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1810,"stamp":"2023-12-05T23:04:03.696717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003594"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":526,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.981305+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:00:38.981305+0000","last_peered":"2023-12-05T23:00:38.981305+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:00:38.981305+0000","last_undegraded":"2023-12-05T23:00:38.981305+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.68300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738893,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:07.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-12-05T23:04:08.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:08.329 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:08.330 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:08.345 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1812,"stamp":"2023-12-05T23:04:07.697884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738894,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.44900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:08.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:08.615 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:08.615 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:08.632 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1812,"stamp":"2023-12-05T23:04:07.697884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738894,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.44900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:08.632 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:08.632 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:08.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:08.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:08.949 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:08.949 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2623,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:08.967 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:08.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:09.237 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:09.238 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:09.253 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1812,"stamp":"2023-12-05T23:04:07.697884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738894,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.44900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:12.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:12.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-05T23:04:12.555 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:12.555 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:12.573 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1814,"stamp":"2023-12-05T23:04:11.699090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738895,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.495}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:12.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:12.848 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:12.848 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:12.864 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1814,"stamp":"2023-12-05T23:04:11.699090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738895,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.495}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:12.865 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:12.865 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:12.865 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:12.865 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:12.865 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:12.865 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:12.866 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:12.866 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:12.866 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:12.866 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:12.866 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:12.866 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:12.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:13.191 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:13.191 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2628,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:13.208 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:13.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:13.474 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:13.474 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:13.493 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1814,"stamp":"2023-12-05T23:04:11.699090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.66000000000000003,"15min":0.59599999999999997},"min":{"1min":0.313,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.69999999999999996,"5min":11.629,"15min":11.629},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.84599999999999997,"15min":0.78600000000000003},"min":{"1min":0.35299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.626,"5min":12.166,"15min":21.992000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738895,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.495}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:16.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:16.841 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:16.842 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:16.860 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1816,"stamp":"2023-12-05T23:04:15.700299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003581"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738896,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.437}]}]},{"osd":1,"up_from":8,"seq":34359738896,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:16.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:17.139 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:17.139 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:17.155 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1816,"stamp":"2023-12-05T23:04:15.700299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003581"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738896,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.437}]}]},{"osd":1,"up_from":8,"seq":34359738896,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:17.156 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:17.156 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:17.156 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:17.156 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:17.157 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:17.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:17.509 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2632,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:17.529 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:17.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:17.792 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:17.792 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:17.808 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1817,"stamp":"2023-12-05T23:04:17.700917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003601"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738896,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:03:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.64200000000000002,"15min":0.69299999999999995},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.58699999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:17.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-12-05T23:04:20.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:21.115 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:21.115 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:21.133 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1818,"stamp":"2023-12-05T23:04:19.701518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003633"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738897,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:21.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:21.413 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:21.414 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:21.430 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1818,"stamp":"2023-12-05T23:04:19.701518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003633"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738897,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:21.431 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:21.431 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:21.431 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:21.431 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:21.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:21.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:21.775 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2636,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:21.794 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:21.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:22.056 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:22.056 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:22.072 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1819,"stamp":"2023-12-05T23:04:21.702157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003685"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":522,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:37.364951+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:00:37.364951+0000","last_peered":"2023-12-05T23:00:37.364951+0000","last_clean":"2023-12-05T23:00:37.364951+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:00:37.364951+0000","last_undegraded":"2023-12-05T23:00:37.364951+0000","last_fullsized":"2023-12-05T23:00:37.364951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738897,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:23.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-12-05T23:04:25.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:25.375 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:25.394 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1820,"stamp":"2023-12-05T23:04:23.702721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003630"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.28599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738897,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:25.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:25.685 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:25.686 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:25.703 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1820,"stamp":"2023-12-05T23:04:23.702721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003630"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.28599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738897,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:25.704 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:25.704 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:25.705 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:25.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:26.049 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2640,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:26.068 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:26.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:26.332 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:26.349 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1821,"stamp":"2023-12-05T23:04:25.703276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003594"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.28599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738898,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:28.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-12-05T23:04:29.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:29.648 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:29.649 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:29.667 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1822,"stamp":"2023-12-05T23:04:27.703907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003608"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:39.387791+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:00:39.387791+0000","last_peered":"2023-12-05T23:00:39.387791+0000","last_clean":"2023-12-05T23:00:39.387791+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:00:39.387791+0000","last_undegraded":"2023-12-05T23:00:39.387791+0000","last_fullsized":"2023-12-05T23:00:39.387791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738898,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:29.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:29.955 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:29.955 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:29.973 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1823,"stamp":"2023-12-05T23:04:29.704512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003595"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:29.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:29.975 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:29.975 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:29.975 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:29.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:30.312 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:30.313 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2645,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:30.330 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:30.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:30.588 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:30.589 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:30.605 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1823,"stamp":"2023-12-05T23:04:29.704512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003595"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:33.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:33.907 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:33.908 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:33.925 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1825,"stamp":"2023-12-05T23:04:33.705708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.372}]}]},{"osd":1,"up_from":8,"seq":34359738899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:33.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:34.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-12-05T23:04:34.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:34.205 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:34.223 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1825,"stamp":"2023-12-05T23:04:33.705708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.372}]}]},{"osd":1,"up_from":8,"seq":34359738899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:34.224 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:34.224 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:34.224 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:34.224 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:34.224 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:34.224 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:34.224 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:34.225 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.851061+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:00:38.851061+0000', 'last_peered': '2023-12-05T23:00:38.851061+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:00:38.851061+0000', 'last_undegraded': '2023-12-05T23:00:38.851061+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:34.225 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:34.225 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:34.225 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:34.225 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:34.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:34.559 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2649,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:34.577 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:34.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:34.848 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:34.848 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:34.865 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1825,"stamp":"2023-12-05T23:04:33.705708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.851061+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:00:38.851061+0000","last_peered":"2023-12-05T23:00:38.851061+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:00:38.851061+0000","last_undegraded":"2023-12-05T23:00:38.851061+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.372}]}]},{"osd":1,"up_from":8,"seq":34359738899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:37.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:38.161 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:38.162 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:38.177 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1827,"stamp":"2023-12-05T23:04:37.706952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003675"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.214},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738900,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.65100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:38.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:38.463 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:38.463 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:38.481 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1827,"stamp":"2023-12-05T23:04:37.706952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003675"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.214},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738900,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.65100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:38.482 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:38.482 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:00:38.957074+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:00:38.957074+0000', 'last_peered': '2023-12-05T23:00:38.957074+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:00:38.957074+0000', 'last_undegraded': '2023-12-05T23:00:38.957074+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:29.461622+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:02:29.461622+0000', 'last_peered': '2023-12-05T23:02:29.461622+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:02:29.461622+0000', 'last_undegraded': '2023-12-05T23:02:29.461622+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:38.483 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:38.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:38.800 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2653,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:38.818 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:38.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:39.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:39.101 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:39.117 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1827,"stamp":"2023-12-05T23:04:37.706952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003675"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:00:38.957074+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:00:38.957074+0000","last_peered":"2023-12-05T23:00:38.957074+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:00:38.957074+0000","last_undegraded":"2023-12-05T23:00:38.957074+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.214},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738900,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.65100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:39.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-12-05T23:04:42.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:42.419 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:42.419 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:42.437 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1829,"stamp":"2023-12-05T23:04:41.708164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003652"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.214},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738901,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.42299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:42.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:42.721 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:42.722 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:42.740 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1829,"stamp":"2023-12-05T23:04:41.708164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003652"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.214},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738901,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.42299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:42.741 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:42.741 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:42.741 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:42.741 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:42.741 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:42.741 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:34.462614+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:02:34.462614+0000', 'last_peered': '2023-12-05T23:02:34.462614+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:02:34.462614+0000', 'last_undegraded': '2023-12-05T23:02:34.462614+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:42.742 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:42.742 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:29.461622+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:02:29.461622+0000', 'last_peered': '2023-12-05T23:02:29.461622+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:02:29.461622+0000', 'last_undegraded': '2023-12-05T23:02:29.461622+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:42.742 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:42.742 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:42.742 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:42.742 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:42.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:43.055 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:43.055 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2657,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:43.072 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:43.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:43.334 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:43.334 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:43.352 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1829,"stamp":"2023-12-05T23:04:41.708164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003652"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.214},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738901,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.42299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:44.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-12-05T23:04:46.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:46.650 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:46.666 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1831,"stamp":"2023-12-05T23:04:45.709345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003636"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.32700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738902,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.57299999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:46.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:46.961 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:46.962 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:46.972 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1831,"stamp":"2023-12-05T23:04:45.709345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003636"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.32700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738902,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.57299999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:46.973 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:46.973 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:34.462614+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:02:34.462614+0000', 'last_peered': '2023-12-05T23:02:34.462614+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:02:34.462614+0000', 'last_undegraded': '2023-12-05T23:02:34.462614+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:29.461622+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:02:29.461622+0000', 'last_peered': '2023-12-05T23:02:29.461622+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:02:29.461622+0000', 'last_undegraded': '2023-12-05T23:02:29.461622+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:46.974 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 658, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:00:49.443185+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:00:49.443185+0000', 'last_peered': '2023-12-05T23:00:49.443185+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:00:49.443185+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:46.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:47.296 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:47.296 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2662,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:47.313 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:47.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:47.585 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:47.585 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:47.601 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1831,"stamp":"2023-12-05T23:04:45.709345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003636"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":658,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:00:49.443185+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:00:49.443185+0000","last_peered":"2023-12-05T23:00:49.443185+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:00:49.443185+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.32700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738902,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.57299999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:50.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-12-05T23:04:50.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:50.903 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:50.922 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1833,"stamp":"2023-12-05T23:04:49.710568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":659,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:02:49.465127+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:02:49.465127+0000","last_peered":"2023-12-05T23:02:49.465127+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:02:49.465127+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738903,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:50.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:51.190 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:51.206 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1833,"stamp":"2023-12-05T23:04:49.710568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":659,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:02:49.465127+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:02:49.465127+0000","last_peered":"2023-12-05T23:02:49.465127+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:02:49.465127+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738903,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:51.207 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:51.207 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:51.207 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:51.207 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:34.462614+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:02:34.462614+0000', 'last_peered': '2023-12-05T23:02:34.462614+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:02:34.462614+0000', 'last_undegraded': '2023-12-05T23:02:34.462614+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:29.461622+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:02:29.461622+0000', 'last_peered': '2023-12-05T23:02:29.461622+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:02:29.461622+0000', 'last_undegraded': '2023-12-05T23:02:29.461622+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:51.208 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 659, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:02:49.465127+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:02:49.465127+0000', 'last_peered': '2023-12-05T23:02:49.465127+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:02:49.465127+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:51.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:51.520 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:51.520 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2666,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:51.538 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:51.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:51.799 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:51.815 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1834,"stamp":"2023-12-05T23:04:51.711172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003566"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":659,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:02:49.465127+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:02:49.465127+0000","last_peered":"2023-12-05T23:02:49.465127+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:02:49.465127+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738903,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:54.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:55.104 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:55.104 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:55.121 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1835,"stamp":"2023-12-05T23:04:53.711746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":659,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:02:49.465127+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:02:49.465127+0000","last_peered":"2023-12-05T23:02:49.465127+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:02:49.465127+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.255},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738903,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:55.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:55.391 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:55.392 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:55.408 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1835,"stamp":"2023-12-05T23:04:53.711746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":659,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:02:49.465127+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:02:49.465127+0000","last_peered":"2023-12-05T23:02:49.465127+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:02:49.465127+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.255},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738903,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:55.409 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-05T23:04:55.409 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.455171+0000', 'last_change': '2023-12-05T22:59:49.820017+0000', 'last_active': '2023-12-05T23:01:54.455171+0000', 'last_peered': '2023-12-05T23:01:54.455171+0000', 'last_clean': '2023-12-05T22:59:46.646257+0000', 'last_became_active': '2023-12-05T22:59:49.820017+0000', 'last_became_peered': '2023-12-05T22:59:49.820017+0000', 'last_unstale': '2023-12-05T23:01:54.455171+0000', 'last_undegraded': '2023-12-05T23:01:54.455171+0000', 'last_fullsized': '2023-12-05T22:59:49.776961+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:55.409 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-05T23:04:55.409 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 527, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:49.453915+0000', 'last_change': '2023-12-05T22:59:49.830049+0000', 'last_active': '2023-12-05T23:01:49.453915+0000', 'last_peered': '2023-12-05T23:01:49.453915+0000', 'last_clean': '2023-12-05T22:59:46.647541+0000', 'last_became_active': '2023-12-05T22:59:49.830049+0000', 'last_became_peered': '2023-12-05T22:59:49.830049+0000', 'last_unstale': '2023-12-05T23:01:49.453915+0000', 'last_undegraded': '2023-12-05T23:01:49.453915+0000', 'last_fullsized': '2023-12-05T22:59:49.789267+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:34.462614+0000', 'last_change': '2023-12-05T22:59:49.835878+0000', 'last_active': '2023-12-05T23:02:34.462614+0000', 'last_peered': '2023-12-05T23:02:34.462614+0000', 'last_clean': '2023-12-05T22:59:46.647432+0000', 'last_became_active': '2023-12-05T22:59:49.835878+0000', 'last_became_peered': '2023-12-05T22:59:49.835878+0000', 'last_unstale': '2023-12-05T23:02:34.462614+0000', 'last_undegraded': '2023-12-05T23:02:34.462614+0000', 'last_fullsized': '2023-12-05T22:59:49.790664+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 528, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:02:29.461622+0000', 'last_change': '2023-12-05T22:59:49.823848+0000', 'last_active': '2023-12-05T23:02:29.461622+0000', 'last_peered': '2023-12-05T23:02:29.461622+0000', 'last_clean': '2023-12-05T22:59:46.642441+0000', 'last_became_active': '2023-12-05T22:59:49.823848+0000', 'last_became_peered': '2023-12-05T22:59:49.823848+0000', 'last_unstale': '2023-12-05T23:02:29.461622+0000', 'last_undegraded': '2023-12-05T23:02:29.461622+0000', 'last_fullsized': '2023-12-05T22:59:49.772897+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2023-12-05T23:01:54.454906+0000', 'last_change': '2023-12-05T22:59:49.839477+0000', 'last_active': '2023-12-05T23:01:54.454906+0000', 'last_peered': '2023-12-05T23:01:54.454906+0000', 'last_clean': '2023-12-05T22:20:41.215240+0000', 'last_became_active': '2023-12-05T22:59:49.839477+0000', 'last_became_peered': '2023-12-05T22:59:49.839477+0000', 'last_unstale': '2023-12-05T23:01:54.454906+0000', 'last_undegraded': '2023-12-05T23:01:54.454906+0000', 'last_fullsized': '2023-12-05T22:59:49.786028+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:41.215240+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-05T23:04:55.410 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "12'97", 'reported_seq': 659, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-05T23:02:49.465127+0000', 'last_change': '2023-12-05T22:59:49.831345+0000', 'last_active': '2023-12-05T23:02:49.465127+0000', 'last_peered': '2023-12-05T23:02:49.465127+0000', 'last_clean': '2023-12-05T22:59:46.642453+0000', 'last_became_active': '2023-12-05T22:59:49.831345+0000', 'last_became_peered': '2023-12-05T22:59:49.831345+0000', 'last_unstale': '2023-12-05T23:02:49.465127+0000', 'last_undegraded': '2023-12-05T22:59:49.799104+0000', 'last_fullsized': '2023-12-05T22:59:49.798824+0000', 'mapping_epoch': 503, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'last_clean_scrub_stamp': '2023-12-05T22:20:37.191172+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-05T23:04:55.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-05T23:04:55.733 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:55.733 INFO:teuthology.orchestra.run.smithi190.stdout:{"fsid":"778655ca-dd08-4388-ac14-e9d094947e15","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2670,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":508,"num_osds":3,"num_up_osds":2,"osd_up_since":1701817188,"num_in_osds":3,"osd_in_since":1701814826,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3632627712,"bytes_avail":286277664768,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-12-05T23:00:43.633310+0000","services":{}},"progress_events":{}} 2023-12-05T23:04:55.750 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-05T23:04:55.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:04:55.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-12-05T23:04:55.757 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-05T23:04:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-12-05T23:04:56.034 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2023-12-05T23:04:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1836,"stamp":"2023-12-05T23:04:55.712350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3547488,"kb_used_data":303024,"kb_used_omap":294,"kb_used_meta":3240025,"kb_avail":279568032,"statfs":{"total":289910292480,"available":286277664768,"internally_reserved":0,"allocated":310296576,"data_stored":300469067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":301344,"internal_metadata":3317786336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003621"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.455171+0000","last_change":"2023-12-05T22:59:49.820017+0000","last_active":"2023-12-05T23:01:54.455171+0000","last_peered":"2023-12-05T23:01:54.455171+0000","last_clean":"2023-12-05T22:59:46.646257+0000","last_became_active":"2023-12-05T22:59:49.820017+0000","last_became_peered":"2023-12-05T22:59:49.820017+0000","last_unstale":"2023-12-05T23:01:54.455171+0000","last_undegraded":"2023-12-05T23:01:54.455171+0000","last_fullsized":"2023-12-05T22:59:49.776961+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:40:13.311898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":527,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:49.453915+0000","last_change":"2023-12-05T22:59:49.830049+0000","last_active":"2023-12-05T23:01:49.453915+0000","last_peered":"2023-12-05T23:01:49.453915+0000","last_clean":"2023-12-05T22:59:46.647541+0000","last_became_active":"2023-12-05T22:59:49.830049+0000","last_became_peered":"2023-12-05T22:59:49.830049+0000","last_unstale":"2023-12-05T23:01:49.453915+0000","last_undegraded":"2023-12-05T23:01:49.453915+0000","last_fullsized":"2023-12-05T22:59:49.789267+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:48:19.892248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:34.462614+0000","last_change":"2023-12-05T22:59:49.835878+0000","last_active":"2023-12-05T23:02:34.462614+0000","last_peered":"2023-12-05T23:02:34.462614+0000","last_clean":"2023-12-05T22:59:46.647432+0000","last_became_active":"2023-12-05T22:59:49.835878+0000","last_became_peered":"2023-12-05T22:59:49.835878+0000","last_unstale":"2023-12-05T23:02:34.462614+0000","last_undegraded":"2023-12-05T23:02:34.462614+0000","last_fullsized":"2023-12-05T22:59:49.790664+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:14:09.732123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:02:29.461622+0000","last_change":"2023-12-05T22:59:49.823848+0000","last_active":"2023-12-05T23:02:29.461622+0000","last_peered":"2023-12-05T23:02:29.461622+0000","last_clean":"2023-12-05T22:59:46.642441+0000","last_became_active":"2023-12-05T22:59:49.823848+0000","last_became_peered":"2023-12-05T22:59:49.823848+0000","last_unstale":"2023-12-05T23:02:29.461622+0000","last_undegraded":"2023-12-05T23:02:29.461622+0000","last_fullsized":"2023-12-05T22:59:49.772897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T08:59:36.718268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":508,"state":"active+undersized","last_fresh":"2023-12-05T23:01:54.454906+0000","last_change":"2023-12-05T22:59:49.839477+0000","last_active":"2023-12-05T23:01:54.454906+0000","last_peered":"2023-12-05T23:01:54.454906+0000","last_clean":"2023-12-05T22:20:41.215240+0000","last_became_active":"2023-12-05T22:59:49.839477+0000","last_became_peered":"2023-12-05T22:59:49.839477+0000","last_unstale":"2023-12-05T23:01:54.454906+0000","last_undegraded":"2023-12-05T23:01:54.454906+0000","last_fullsized":"2023-12-05T22:59:49.786028+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:23:03.103043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:17.373554+0000","last_change":"2023-12-05T22:20:42.251237+0000","last_active":"2023-12-05T23:02:17.373554+0000","last_peered":"2023-12-05T23:02:17.373554+0000","last_clean":"2023-12-05T23:02:17.373554+0000","last_became_active":"2023-12-05T22:20:42.250325+0000","last_became_peered":"2023-12-05T22:20:42.250325+0000","last_unstale":"2023-12-05T23:02:17.373554+0000","last_undegraded":"2023-12-05T23:02:17.373554+0000","last_fullsized":"2023-12-05T23:02:17.373554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:44:31.180626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:00:57.361629+0000","last_change":"2023-12-05T22:20:42.248911+0000","last_active":"2023-12-05T23:00:57.361629+0000","last_peered":"2023-12-05T23:00:57.361629+0000","last_clean":"2023-12-05T23:00:57.361629+0000","last_became_active":"2023-12-05T22:20:42.248423+0000","last_became_peered":"2023-12-05T22:20:42.248423+0000","last_unstale":"2023-12-05T23:00:57.361629+0000","last_undegraded":"2023-12-05T23:00:57.361629+0000","last_fullsized":"2023-12-05T23:00:57.361629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:29:21.016637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":524,"reported_epoch":508,"state":"active+clean","last_fresh":"2023-12-05T23:02:24.460541+0000","last_change":"2023-12-05T22:20:42.247477+0000","last_active":"2023-12-05T23:02:24.460541+0000","last_peered":"2023-12-05T23:02:24.460541+0000","last_clean":"2023-12-05T23:02:24.460541+0000","last_became_active":"2023-12-05T22:20:42.247038+0000","last_became_peered":"2023-12-05T22:20:42.247038+0000","last_unstale":"2023-12-05T23:02:24.460541+0000","last_undegraded":"2023-12-05T23:02:24.460541+0000","last_fullsized":"2023-12-05T23:02:24.460541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:41.215240+0000","last_clean_scrub_stamp":"2023-12-05T22:20:41.215240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T06:29:36.071469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":659,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2023-12-05T23:02:49.465127+0000","last_change":"2023-12-05T22:59:49.831345+0000","last_active":"2023-12-05T23:02:49.465127+0000","last_peered":"2023-12-05T23:02:49.465127+0000","last_clean":"2023-12-05T22:59:46.642453+0000","last_became_active":"2023-12-05T22:59:49.831345+0000","last_became_peered":"2023-12-05T22:59:49.831345+0000","last_unstale":"2023-12-05T23:02:49.465127+0000","last_undegraded":"2023-12-05T22:59:49.799104+0000","last_fullsized":"2023-12-05T22:59:49.798824+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:37.191172+0000","last_clean_scrub_stamp":"2023-12-05T22:20:37.191172+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:57:19.177089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937944,"kb_used_data":5900,"kb_used_omap":92,"kb_used_meta":931939,"kb_avail":93433896,"statfs":{"total":96636764160,"available":95676309504,"internally_reserved":0,"allocated":6041600,"data_stored":3017160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94658,"internal_metadata":954306110},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:02 2023","interfaces":[{"interface":"back","average":{"1min":2.3100000000000001,"5min":0.93400000000000005,"15min":0.752},"min":{"1min":0.27600000000000002,"5min":0.189,"15min":0.156},"max":{"1min":35.996000000000002,"5min":35.996000000000002,"15min":35.996000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.3969999999999998,"5min":0.95799999999999996,"15min":0.754},"min":{"1min":0.24299999999999999,"5min":0.187,"15min":0.187},"max":{"1min":35.904000000000003,"5min":35.904000000000003,"15min":35.904000000000003},"last":2.04}]},{"osd":1,"last update":"Tue Dec 5 23:04:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.186,"15min":0.17100000000000001},"max":{"1min":0.58799999999999997,"5min":11.629,"15min":11.629},"last":0.255},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.65100000000000002,"15min":0.78200000000000003},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.68300000000000005,"5min":11.593,"15min":21.992000000000001},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738904,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1042908,"kb_used_data":6480,"kb_used_omap":89,"kb_used_meta":1036326,"kb_avail":93328932,"statfs":{"total":96636764160,"available":95568826368,"internally_reserved":0,"allocated":6635520,"data_stored":3607528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91574,"internal_metadata":1061198410},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 22:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.766,"5min":2.8999999999999999,"15min":6.5190000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.68700000000000006,"15min":0.65900000000000003},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.7330000000000001,"5min":4.1529999999999996,"15min":6.3540000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Dec 5 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.62,"15min":0.68899999999999995},"min":{"1min":0.28499999999999998,"5min":0.186,"15min":0.186},"max":{"1min":0.59599999999999997,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.438},{"interface":"front","average":{"1min":0.499,"5min":0.63,"15min":0.626},"min":{"1min":0.32600000000000001,"5min":0.22,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":8.8119999999999994,"15min":8.8119999999999994},"last":0.50900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1566636,"kb_used_data":290644,"kb_used_omap":112,"kb_used_meta":1271759,"kb_avail":92805204,"statfs":{"total":96636764160,"available":95032528896,"internally_reserved":0,"allocated":297619456,"data_stored":293844379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":115112,"internal_metadata":1302281816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.79300000000000004,"15min":0.61899999999999999},"min":{"1min":0.17499999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.80500000000000005,"5min":13.914999999999999,"15min":13.914999999999999},"last":1.417},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.90400000000000003,"15min":0.73199999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.3149999999999999,"5min":12.1,"15min":12.1},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Dec 5 22:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85699999999999998,"15min":0.70299999999999996},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.6040000000000001,"5min":11.667999999999999,"15min":11.667999999999999},"last":5.8650000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.92900000000000005,"15min":0.71499999999999997},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.20000000000000001},"max":{"1min":2.621,"5min":11.723000000000001,"15min":11.723000000000001},"last":5.8319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:04:56.161 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-12-05T23:04:56.161 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-05T23:04:56.162 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-05T23:04:56.162 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-05T23:04:56.163 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T23:04:56.168+0000 7fa64bd15640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 35032) UID: 0 2023-12-05T23:04:56.163 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T23:04:56.168+0000 7fa64bd15640 -1 osd.1 508 *** Got signal Terminated *** 2023-12-05T23:04:56.163 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2023-12-05T23:04:56.168+0000 7fa64bd15640 -1 osd.1 508 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-05T23:04:56.164 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T23:04:56.168+0000 7fe8fd8e7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 35034) UID: 0 2023-12-05T23:04:56.164 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T23:04:56.168+0000 7fe8fd8e7640 -1 osd.2 508 *** Got signal Terminated *** 2023-12-05T23:04:56.164 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2023-12-05T23:04:56.168+0000 7fe8fd8e7640 -1 osd.2 508 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-05T23:04:56.164 INFO:tasks.ceph.cluster2.mon.a.smithi190.stderr:2023-12-05T23:04:56.168+0000 7f8aa8742640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 34884) UID: 0 2023-12-05T23:04:56.165 INFO:tasks.ceph.cluster2.mon.a.smithi190.stderr:2023-12-05T23:04:56.168+0000 7f8aa8742640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-05T23:04:56.165 INFO:tasks.ceph.cluster2.mgr.x.smithi062.stderr:2023-12-05T23:04:56.165+0000 7fb1b1a8f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 40003) UID: 0 2023-12-05T23:04:59.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-05T23:06:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-05T23:06:59.078 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-05T23:06:59.081 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-05T23:06:59.081 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-05T23:06:59.081 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-12-05T23:06:59.081 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:06:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-05T23:06:59.081 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-12-05T23:06:59.082 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-12-05T23:06:59.082 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-12-05T23:06:59.082 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:06:59.082 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-12-05T23:06:59.082 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-12-05T23:06:59.082 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:06:59.082 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-12-05T23:06:59.083 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-05T23:06:59.083 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-12-05T23:06:59.083 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:06:59.083 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-12-05T23:06:59.083 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-05T23:06:59.083 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-12-05T23:06:59.083 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:06:59.083 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-12-05T23:06:59.084 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-05T23:06:59.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-05T23:06:59.100 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T23:06:59.100 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-12-05T23:06:59.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T23:06:59.405 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-12-05T23:06:59.810 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T23:06:59.811 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-12-05T23:07:00.079 INFO:tasks.ceph:Archiving mon data... 2023-12-05T23:07:00.081 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/data/cluster2.mon.a.tgz 2023-12-05T23:07:00.081 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2023-12-05T23:07:00.086 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.AeY0aYhKvh 2023-12-05T23:07:00.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.AeY0aYhKvh 2023-12-05T23:07:00.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.AeY0aYhKvh 2023-12-05T23:07:00.291 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.AeY0aYhKvh is 404KB 2023-12-05T23:07:00.358 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.AeY0aYhKvh 2023-12-05T23:07:00.362 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-05T23:07:00.362 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-05T23:07:00.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-05T23:07:00.418 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-05T23:07:00.418 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-05T23:07:00.431 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-05T23:07:00.432 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-05T23:07:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T23:07:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2023-12-05T23:07:00.757 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":1880,"stamp":"2023-12-05T23:06:59.341601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":444612,"kb_used_data":12512,"kb_used_omap":4,"kb_used_meta":432059,"kb_avail":282670908,"statfs":{"total":289910292480,"available":289455009792,"internally_reserved":0,"allocated":12812288,"data_stored":7405457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":442428751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002994"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453817+0000","last_change":"2023-12-05T22:20:03.947467+0000","last_active":"2023-12-05T23:03:23.453817+0000","last_peered":"2023-12-05T23:03:23.453817+0000","last_clean":"2023-12-05T23:03:23.453817+0000","last_became_active":"2023-12-05T22:20:03.947069+0000","last_became_peered":"2023-12-05T22:20:03.947069+0000","last_unstale":"2023-12-05T23:03:23.453817+0000","last_undegraded":"2023-12-05T23:03:23.453817+0000","last_fullsized":"2023-12-05T23:03:23.453817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:30:10.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453784+0000","last_change":"2023-12-05T22:20:03.955027+0000","last_active":"2023-12-05T23:03:23.453784+0000","last_peered":"2023-12-05T23:03:23.453784+0000","last_clean":"2023-12-05T23:03:23.453784+0000","last_became_active":"2023-12-05T22:20:03.954510+0000","last_became_peered":"2023-12-05T22:20:03.954510+0000","last_unstale":"2023-12-05T23:03:23.453784+0000","last_undegraded":"2023-12-05T23:03:23.453784+0000","last_fullsized":"2023-12-05T23:03:23.453784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:23:12.821704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:58.459554+0000","last_change":"2023-12-05T22:20:03.955392+0000","last_active":"2023-12-05T23:03:58.459554+0000","last_peered":"2023-12-05T23:03:58.459554+0000","last_clean":"2023-12-05T23:03:58.459554+0000","last_became_active":"2023-12-05T22:20:03.954778+0000","last_became_peered":"2023-12-05T22:20:03.954778+0000","last_unstale":"2023-12-05T23:03:58.459554+0000","last_undegraded":"2023-12-05T23:03:58.459554+0000","last_fullsized":"2023-12-05T23:03:58.459554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:49:04.013308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458679+0000","last_change":"2023-12-05T22:20:03.954927+0000","last_active":"2023-12-05T23:03:53.458679+0000","last_peered":"2023-12-05T23:03:53.458679+0000","last_clean":"2023-12-05T23:03:53.458679+0000","last_became_active":"2023-12-05T22:20:03.954437+0000","last_became_peered":"2023-12-05T22:20:03.954437+0000","last_unstale":"2023-12-05T23:03:53.458679+0000","last_undegraded":"2023-12-05T23:03:53.458679+0000","last_fullsized":"2023-12-05T23:03:53.458679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:36:37.288695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.334539+0000","last_change":"2023-12-05T22:20:03.949404+0000","last_active":"2023-12-05T23:03:48.334539+0000","last_peered":"2023-12-05T23:03:48.334539+0000","last_clean":"2023-12-05T23:03:48.334539+0000","last_became_active":"2023-12-05T22:20:03.948733+0000","last_became_peered":"2023-12-05T22:20:03.948733+0000","last_unstale":"2023-12-05T23:03:48.334539+0000","last_undegraded":"2023-12-05T23:03:48.334539+0000","last_fullsized":"2023-12-05T23:03:48.334539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:35:07.902401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.399226+0000","last_change":"2023-12-05T22:20:03.950953+0000","last_active":"2023-12-05T23:03:48.399226+0000","last_peered":"2023-12-05T23:03:48.399226+0000","last_clean":"2023-12-05T23:03:48.399226+0000","last_became_active":"2023-12-05T22:20:03.950488+0000","last_became_peered":"2023-12-05T22:20:03.950488+0000","last_unstale":"2023-12-05T23:03:48.399226+0000","last_undegraded":"2023-12-05T23:03:48.399226+0000","last_fullsized":"2023-12-05T23:03:48.399226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:40:00.287834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:13.406015+0000","last_change":"2023-12-05T22:20:03.948835+0000","last_active":"2023-12-05T23:04:13.406015+0000","last_peered":"2023-12-05T23:04:13.406015+0000","last_clean":"2023-12-05T23:04:13.406015+0000","last_became_active":"2023-12-05T22:20:03.948291+0000","last_became_peered":"2023-12-05T22:20:03.948291+0000","last_unstale":"2023-12-05T23:04:13.406015+0000","last_undegraded":"2023-12-05T23:04:13.406015+0000","last_fullsized":"2023-12-05T23:04:13.406015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:00:52.805869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458627+0000","last_change":"2023-12-05T22:20:03.944949+0000","last_active":"2023-12-05T23:03:53.458627+0000","last_peered":"2023-12-05T23:03:53.458627+0000","last_clean":"2023-12-05T23:03:53.458627+0000","last_became_active":"2023-12-05T22:20:03.944294+0000","last_became_peered":"2023-12-05T22:20:03.944294+0000","last_unstale":"2023-12-05T23:03:53.458627+0000","last_undegraded":"2023-12-05T23:03:53.458627+0000","last_fullsized":"2023-12-05T23:03:53.458627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:52:50.330306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":619,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:08.461143+0000","last_change":"2023-12-05T22:20:00.475423+0000","last_active":"2023-12-05T23:04:08.461143+0000","last_peered":"2023-12-05T23:04:08.461143+0000","last_clean":"2023-12-05T23:04:08.461143+0000","last_became_active":"2023-12-05T22:20:00.473094+0000","last_became_peered":"2023-12-05T22:20:00.473094+0000","last_unstale":"2023-12-05T23:04:08.461143+0000","last_undegraded":"2023-12-05T23:04:08.461143+0000","last_fullsized":"2023-12-05T23:04:08.461143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_clean_scrub_stamp":"2023-12-05T22:19:58.883597+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:32:18.798306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147348,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94224492,"statfs":{"total":96636764160,"available":96485879808,"internally_reserved":0,"allocated":3874816,"data_stored":2074907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53700000000000003,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.21299999999999999},"max":{"1min":0.65500000000000003,"5min":0.86399999999999999,"15min":2.6389999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.621,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.23400000000000001},"max":{"1min":0.68000000000000005,"5min":0.81999999999999995,"15min":2.6779999999999999},"last":0.68999999999999995}]},{"osd":1,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.47399999999999998,"5min":0.46100000000000002,"15min":0.19},"max":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":1.3580000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67100000000000004,"15min":0.69599999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.26800000000000002},"max":{"1min":0.77300000000000002,"5min":0.90500000000000003,"15min":5.8959999999999999},"last":0.64600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738937,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121560,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250280,"statfs":{"total":96636764160,"available":96512286720,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56000000000000005,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.20300000000000001},"max":{"1min":0.73299999999999998,"5min":0.754,"15min":13.76},"last":0.625},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.249},"max":{"1min":0.66900000000000004,"5min":0.73399999999999999,"15min":8.4659999999999993},"last":0.54500000000000004}]},{"osd":2,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.622,"15min":0.66000000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.29599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":8.2270000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63500000000000001,"15min":0.69499999999999995},"min":{"1min":0.50900000000000001,"5min":0.48499999999999999,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":14.518000000000001},"last":0.74099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175704,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196136,"statfs":{"total":96636764160,"available":96456843264,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.214},"max":{"1min":0.69099999999999995,"5min":0.749,"15min":2.9390000000000001},"last":0.498},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.50900000000000001,"5min":0.495,"15min":0.23200000000000001},"max":{"1min":0.68100000000000005,"5min":0.78100000000000003,"15min":11.662000000000001},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.27100000000000002},"max":{"1min":0.79100000000000004,"5min":0.89600000000000002,"15min":2.714},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.29799999999999999},"max":{"1min":0.78400000000000003,"5min":0.96599999999999997,"15min":2.843},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:07:00.759 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-05T23:07:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T23:07:01.053 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2023-12-05T23:07:01.071 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":1880,"stamp":"2023-12-05T23:06:59.341601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":444612,"kb_used_data":12512,"kb_used_omap":4,"kb_used_meta":432059,"kb_avail":282670908,"statfs":{"total":289910292480,"available":289455009792,"internally_reserved":0,"allocated":12812288,"data_stored":7405457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":442428751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002994"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453817+0000","last_change":"2023-12-05T22:20:03.947467+0000","last_active":"2023-12-05T23:03:23.453817+0000","last_peered":"2023-12-05T23:03:23.453817+0000","last_clean":"2023-12-05T23:03:23.453817+0000","last_became_active":"2023-12-05T22:20:03.947069+0000","last_became_peered":"2023-12-05T22:20:03.947069+0000","last_unstale":"2023-12-05T23:03:23.453817+0000","last_undegraded":"2023-12-05T23:03:23.453817+0000","last_fullsized":"2023-12-05T23:03:23.453817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:30:10.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453784+0000","last_change":"2023-12-05T22:20:03.955027+0000","last_active":"2023-12-05T23:03:23.453784+0000","last_peered":"2023-12-05T23:03:23.453784+0000","last_clean":"2023-12-05T23:03:23.453784+0000","last_became_active":"2023-12-05T22:20:03.954510+0000","last_became_peered":"2023-12-05T22:20:03.954510+0000","last_unstale":"2023-12-05T23:03:23.453784+0000","last_undegraded":"2023-12-05T23:03:23.453784+0000","last_fullsized":"2023-12-05T23:03:23.453784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:23:12.821704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:58.459554+0000","last_change":"2023-12-05T22:20:03.955392+0000","last_active":"2023-12-05T23:03:58.459554+0000","last_peered":"2023-12-05T23:03:58.459554+0000","last_clean":"2023-12-05T23:03:58.459554+0000","last_became_active":"2023-12-05T22:20:03.954778+0000","last_became_peered":"2023-12-05T22:20:03.954778+0000","last_unstale":"2023-12-05T23:03:58.459554+0000","last_undegraded":"2023-12-05T23:03:58.459554+0000","last_fullsized":"2023-12-05T23:03:58.459554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:49:04.013308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458679+0000","last_change":"2023-12-05T22:20:03.954927+0000","last_active":"2023-12-05T23:03:53.458679+0000","last_peered":"2023-12-05T23:03:53.458679+0000","last_clean":"2023-12-05T23:03:53.458679+0000","last_became_active":"2023-12-05T22:20:03.954437+0000","last_became_peered":"2023-12-05T22:20:03.954437+0000","last_unstale":"2023-12-05T23:03:53.458679+0000","last_undegraded":"2023-12-05T23:03:53.458679+0000","last_fullsized":"2023-12-05T23:03:53.458679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:36:37.288695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.334539+0000","last_change":"2023-12-05T22:20:03.949404+0000","last_active":"2023-12-05T23:03:48.334539+0000","last_peered":"2023-12-05T23:03:48.334539+0000","last_clean":"2023-12-05T23:03:48.334539+0000","last_became_active":"2023-12-05T22:20:03.948733+0000","last_became_peered":"2023-12-05T22:20:03.948733+0000","last_unstale":"2023-12-05T23:03:48.334539+0000","last_undegraded":"2023-12-05T23:03:48.334539+0000","last_fullsized":"2023-12-05T23:03:48.334539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:35:07.902401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.399226+0000","last_change":"2023-12-05T22:20:03.950953+0000","last_active":"2023-12-05T23:03:48.399226+0000","last_peered":"2023-12-05T23:03:48.399226+0000","last_clean":"2023-12-05T23:03:48.399226+0000","last_became_active":"2023-12-05T22:20:03.950488+0000","last_became_peered":"2023-12-05T22:20:03.950488+0000","last_unstale":"2023-12-05T23:03:48.399226+0000","last_undegraded":"2023-12-05T23:03:48.399226+0000","last_fullsized":"2023-12-05T23:03:48.399226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:40:00.287834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:13.406015+0000","last_change":"2023-12-05T22:20:03.948835+0000","last_active":"2023-12-05T23:04:13.406015+0000","last_peered":"2023-12-05T23:04:13.406015+0000","last_clean":"2023-12-05T23:04:13.406015+0000","last_became_active":"2023-12-05T22:20:03.948291+0000","last_became_peered":"2023-12-05T22:20:03.948291+0000","last_unstale":"2023-12-05T23:04:13.406015+0000","last_undegraded":"2023-12-05T23:04:13.406015+0000","last_fullsized":"2023-12-05T23:04:13.406015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:00:52.805869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458627+0000","last_change":"2023-12-05T22:20:03.944949+0000","last_active":"2023-12-05T23:03:53.458627+0000","last_peered":"2023-12-05T23:03:53.458627+0000","last_clean":"2023-12-05T23:03:53.458627+0000","last_became_active":"2023-12-05T22:20:03.944294+0000","last_became_peered":"2023-12-05T22:20:03.944294+0000","last_unstale":"2023-12-05T23:03:53.458627+0000","last_undegraded":"2023-12-05T23:03:53.458627+0000","last_fullsized":"2023-12-05T23:03:53.458627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:52:50.330306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":619,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:08.461143+0000","last_change":"2023-12-05T22:20:00.475423+0000","last_active":"2023-12-05T23:04:08.461143+0000","last_peered":"2023-12-05T23:04:08.461143+0000","last_clean":"2023-12-05T23:04:08.461143+0000","last_became_active":"2023-12-05T22:20:00.473094+0000","last_became_peered":"2023-12-05T22:20:00.473094+0000","last_unstale":"2023-12-05T23:04:08.461143+0000","last_undegraded":"2023-12-05T23:04:08.461143+0000","last_fullsized":"2023-12-05T23:04:08.461143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_clean_scrub_stamp":"2023-12-05T22:19:58.883597+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:32:18.798306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147348,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94224492,"statfs":{"total":96636764160,"available":96485879808,"internally_reserved":0,"allocated":3874816,"data_stored":2074907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53700000000000003,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.21299999999999999},"max":{"1min":0.65500000000000003,"5min":0.86399999999999999,"15min":2.6389999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.621,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.23400000000000001},"max":{"1min":0.68000000000000005,"5min":0.81999999999999995,"15min":2.6779999999999999},"last":0.68999999999999995}]},{"osd":1,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.47399999999999998,"5min":0.46100000000000002,"15min":0.19},"max":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":1.3580000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67100000000000004,"15min":0.69599999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.26800000000000002},"max":{"1min":0.77300000000000002,"5min":0.90500000000000003,"15min":5.8959999999999999},"last":0.64600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738937,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121560,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250280,"statfs":{"total":96636764160,"available":96512286720,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56000000000000005,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.20300000000000001},"max":{"1min":0.73299999999999998,"5min":0.754,"15min":13.76},"last":0.625},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.249},"max":{"1min":0.66900000000000004,"5min":0.73399999999999999,"15min":8.4659999999999993},"last":0.54500000000000004}]},{"osd":2,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.622,"15min":0.66000000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.29599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":8.2270000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63500000000000001,"15min":0.69499999999999995},"min":{"1min":0.50900000000000001,"5min":0.48499999999999999,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":14.518000000000001},"last":0.74099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175704,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196136,"statfs":{"total":96636764160,"available":96456843264,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.214},"max":{"1min":0.69099999999999995,"5min":0.749,"15min":2.9390000000000001},"last":0.498},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.50900000000000001,"5min":0.495,"15min":0.23200000000000001},"max":{"1min":0.68100000000000005,"5min":0.78100000000000003,"15min":11.662000000000001},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.27100000000000002},"max":{"1min":0.79100000000000004,"5min":0.89600000000000002,"15min":2.714},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.29799999999999999},"max":{"1min":0.78400000000000003,"5min":0.96599999999999997,"15min":2.843},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:07:01.072 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-05T23:07:01.073 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-05T23:07:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T23:07:01.355 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2023-12-05T23:07:01.371 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":1881,"stamp":"2023-12-05T23:07:01.342157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":444612,"kb_used_data":12512,"kb_used_omap":4,"kb_used_meta":432059,"kb_avail":282670908,"statfs":{"total":289910292480,"available":289455009792,"internally_reserved":0,"allocated":12812288,"data_stored":7405457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":442428751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003051"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453817+0000","last_change":"2023-12-05T22:20:03.947467+0000","last_active":"2023-12-05T23:03:23.453817+0000","last_peered":"2023-12-05T23:03:23.453817+0000","last_clean":"2023-12-05T23:03:23.453817+0000","last_became_active":"2023-12-05T22:20:03.947069+0000","last_became_peered":"2023-12-05T22:20:03.947069+0000","last_unstale":"2023-12-05T23:03:23.453817+0000","last_undegraded":"2023-12-05T23:03:23.453817+0000","last_fullsized":"2023-12-05T23:03:23.453817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:30:10.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453784+0000","last_change":"2023-12-05T22:20:03.955027+0000","last_active":"2023-12-05T23:03:23.453784+0000","last_peered":"2023-12-05T23:03:23.453784+0000","last_clean":"2023-12-05T23:03:23.453784+0000","last_became_active":"2023-12-05T22:20:03.954510+0000","last_became_peered":"2023-12-05T22:20:03.954510+0000","last_unstale":"2023-12-05T23:03:23.453784+0000","last_undegraded":"2023-12-05T23:03:23.453784+0000","last_fullsized":"2023-12-05T23:03:23.453784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:23:12.821704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:58.459554+0000","last_change":"2023-12-05T22:20:03.955392+0000","last_active":"2023-12-05T23:03:58.459554+0000","last_peered":"2023-12-05T23:03:58.459554+0000","last_clean":"2023-12-05T23:03:58.459554+0000","last_became_active":"2023-12-05T22:20:03.954778+0000","last_became_peered":"2023-12-05T22:20:03.954778+0000","last_unstale":"2023-12-05T23:03:58.459554+0000","last_undegraded":"2023-12-05T23:03:58.459554+0000","last_fullsized":"2023-12-05T23:03:58.459554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:49:04.013308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458679+0000","last_change":"2023-12-05T22:20:03.954927+0000","last_active":"2023-12-05T23:03:53.458679+0000","last_peered":"2023-12-05T23:03:53.458679+0000","last_clean":"2023-12-05T23:03:53.458679+0000","last_became_active":"2023-12-05T22:20:03.954437+0000","last_became_peered":"2023-12-05T22:20:03.954437+0000","last_unstale":"2023-12-05T23:03:53.458679+0000","last_undegraded":"2023-12-05T23:03:53.458679+0000","last_fullsized":"2023-12-05T23:03:53.458679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:36:37.288695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.334539+0000","last_change":"2023-12-05T22:20:03.949404+0000","last_active":"2023-12-05T23:03:48.334539+0000","last_peered":"2023-12-05T23:03:48.334539+0000","last_clean":"2023-12-05T23:03:48.334539+0000","last_became_active":"2023-12-05T22:20:03.948733+0000","last_became_peered":"2023-12-05T22:20:03.948733+0000","last_unstale":"2023-12-05T23:03:48.334539+0000","last_undegraded":"2023-12-05T23:03:48.334539+0000","last_fullsized":"2023-12-05T23:03:48.334539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:35:07.902401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.399226+0000","last_change":"2023-12-05T22:20:03.950953+0000","last_active":"2023-12-05T23:03:48.399226+0000","last_peered":"2023-12-05T23:03:48.399226+0000","last_clean":"2023-12-05T23:03:48.399226+0000","last_became_active":"2023-12-05T22:20:03.950488+0000","last_became_peered":"2023-12-05T22:20:03.950488+0000","last_unstale":"2023-12-05T23:03:48.399226+0000","last_undegraded":"2023-12-05T23:03:48.399226+0000","last_fullsized":"2023-12-05T23:03:48.399226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:40:00.287834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:13.406015+0000","last_change":"2023-12-05T22:20:03.948835+0000","last_active":"2023-12-05T23:04:13.406015+0000","last_peered":"2023-12-05T23:04:13.406015+0000","last_clean":"2023-12-05T23:04:13.406015+0000","last_became_active":"2023-12-05T22:20:03.948291+0000","last_became_peered":"2023-12-05T22:20:03.948291+0000","last_unstale":"2023-12-05T23:04:13.406015+0000","last_undegraded":"2023-12-05T23:04:13.406015+0000","last_fullsized":"2023-12-05T23:04:13.406015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:00:52.805869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458627+0000","last_change":"2023-12-05T22:20:03.944949+0000","last_active":"2023-12-05T23:03:53.458627+0000","last_peered":"2023-12-05T23:03:53.458627+0000","last_clean":"2023-12-05T23:03:53.458627+0000","last_became_active":"2023-12-05T22:20:03.944294+0000","last_became_peered":"2023-12-05T22:20:03.944294+0000","last_unstale":"2023-12-05T23:03:53.458627+0000","last_undegraded":"2023-12-05T23:03:53.458627+0000","last_fullsized":"2023-12-05T23:03:53.458627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:52:50.330306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":619,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:08.461143+0000","last_change":"2023-12-05T22:20:00.475423+0000","last_active":"2023-12-05T23:04:08.461143+0000","last_peered":"2023-12-05T23:04:08.461143+0000","last_clean":"2023-12-05T23:04:08.461143+0000","last_became_active":"2023-12-05T22:20:00.473094+0000","last_became_peered":"2023-12-05T22:20:00.473094+0000","last_unstale":"2023-12-05T23:04:08.461143+0000","last_undegraded":"2023-12-05T23:04:08.461143+0000","last_fullsized":"2023-12-05T23:04:08.461143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_clean_scrub_stamp":"2023-12-05T22:19:58.883597+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:32:18.798306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147348,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94224492,"statfs":{"total":96636764160,"available":96485879808,"internally_reserved":0,"allocated":3874816,"data_stored":2074907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53700000000000003,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.21299999999999999},"max":{"1min":0.65500000000000003,"5min":0.86399999999999999,"15min":2.6389999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.621,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.23400000000000001},"max":{"1min":0.68000000000000005,"5min":0.81999999999999995,"15min":2.6779999999999999},"last":0.68999999999999995}]},{"osd":1,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.47399999999999998,"5min":0.46100000000000002,"15min":0.19},"max":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":1.3580000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67100000000000004,"15min":0.69599999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.26800000000000002},"max":{"1min":0.77300000000000002,"5min":0.90500000000000003,"15min":5.8959999999999999},"last":0.64600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738937,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121560,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250280,"statfs":{"total":96636764160,"available":96512286720,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56000000000000005,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.20300000000000001},"max":{"1min":0.73299999999999998,"5min":0.754,"15min":13.76},"last":0.625},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.249},"max":{"1min":0.66900000000000004,"5min":0.73399999999999999,"15min":8.4659999999999993},"last":0.54500000000000004}]},{"osd":2,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.622,"15min":0.66000000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.29599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":8.2270000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63500000000000001,"15min":0.69499999999999995},"min":{"1min":0.50900000000000001,"5min":0.48499999999999999,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":14.518000000000001},"last":0.74099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175704,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196136,"statfs":{"total":96636764160,"available":96456843264,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.214},"max":{"1min":0.69099999999999995,"5min":0.749,"15min":2.9390000000000001},"last":0.498},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.50900000000000001,"5min":0.495,"15min":0.23200000000000001},"max":{"1min":0.68100000000000005,"5min":0.78100000000000003,"15min":11.662000000000001},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.27100000000000002},"max":{"1min":0.79100000000000004,"5min":0.89600000000000002,"15min":2.714},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.29799999999999999},"max":{"1min":0.78400000000000003,"5min":0.96599999999999997,"15min":2.843},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:07:01.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-05T23:07:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T23:07:01.646 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":471,"fsid":"c638f251-f7eb-46b0-9628-a265a578cb3d","created":"2023-12-05T22:19:46.901568+0000","modified":"2023-12-05T23:00:34.317842+0000","last_up_change":"2023-12-05T22:19:56.877430+0000","last_in_change":"2023-12-05T22:19:47.820370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-05T22:19:58.427473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-05T22:20:02.290443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"15e3053e-fa26-40a9-958f-4e09c36e0052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6805","nonce":3557713586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6811","nonce":3557713586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6815","nonce":3557713586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3557713586},{"type":"v1","addr":"172.21.15.62:6813","nonce":3557713586}]},"public_addr":"172.21.15.62:6805/3557713586","cluster_addr":"172.21.15.62:6811/3557713586","heartbeat_back_addr":"172.21.15.62:6815/3557713586","heartbeat_front_addr":"172.21.15.62:6813/3557713586","state":["exists","up"]},{"osd":1,"uuid":"92c57430-c95d-44c9-887f-2f80c09752b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6801","nonce":203106146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6803","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6804","nonce":203106146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6810","nonce":203106146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":203106146},{"type":"v1","addr":"172.21.15.62:6807","nonce":203106146}]},"public_addr":"172.21.15.62:6801/203106146","cluster_addr":"172.21.15.62:6804/203106146","heartbeat_back_addr":"172.21.15.62:6810/203106146","heartbeat_front_addr":"172.21.15.62:6807/203106146","state":["exists","up"]},{"osd":2,"uuid":"641eb594-1137-4504-a975-626c541f33bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6817","nonce":3043463782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6819","nonce":3043463782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6823","nonce":3043463782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3043463782},{"type":"v1","addr":"172.21.15.62:6821","nonce":3043463782}]},"public_addr":"172.21.15.62:6817/3043463782","cluster_addr":"172.21.15.62:6819/3043463782","heartbeat_back_addr":"172.21.15.62:6823/3043463782","heartbeat_front_addr":"172.21.15.62:6821/3043463782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.334766+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.341700+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-05T22:19:53.291465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":386,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-05T23:07:02.663 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2023-12-05T23:07:02.664 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-05T23:07:02.828 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2023-12-05T23:07:02.829 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-05T23:07:02.829 INFO:teuthology.orchestra.run.smithi062.stdout:} 2023-12-05T23:07:02.845 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2023-12-05T23:07:03.133 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 0 to deep-scrub 2023-12-05T23:07:03.152 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2023-12-05T23:07:03.152 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-05T23:07:03.295 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2023-12-05T23:07:03.295 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-05T23:07:03.295 INFO:teuthology.orchestra.run.smithi062.stdout:} 2023-12-05T23:07:03.311 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2023-12-05T23:07:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 1 to deep-scrub 2023-12-05T23:07:03.616 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2023-12-05T23:07:03.617 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-05T23:07:03.767 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2023-12-05T23:07:03.767 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-05T23:07:03.767 INFO:teuthology.orchestra.run.smithi062.stdout:} 2023-12-05T23:07:03.783 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2023-12-05T23:07:04.056 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 2 to deep-scrub 2023-12-05T23:07:04.073 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-05T23:07:04.353 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T23:07:04.353 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2023-12-05T23:07:04.371 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":1882,"stamp":"2023-12-05T23:07:03.342590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":444612,"kb_used_data":12512,"kb_used_omap":4,"kb_used_meta":432059,"kb_avail":282670908,"statfs":{"total":289910292480,"available":289455009792,"internally_reserved":0,"allocated":12812288,"data_stored":7405457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":442428751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002958"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453817+0000","last_change":"2023-12-05T22:20:03.947467+0000","last_active":"2023-12-05T23:03:23.453817+0000","last_peered":"2023-12-05T23:03:23.453817+0000","last_clean":"2023-12-05T23:03:23.453817+0000","last_became_active":"2023-12-05T22:20:03.947069+0000","last_became_peered":"2023-12-05T22:20:03.947069+0000","last_unstale":"2023-12-05T23:03:23.453817+0000","last_undegraded":"2023-12-05T23:03:23.453817+0000","last_fullsized":"2023-12-05T23:03:23.453817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:30:10.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":487,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:23.453784+0000","last_change":"2023-12-05T22:20:03.955027+0000","last_active":"2023-12-05T23:03:23.453784+0000","last_peered":"2023-12-05T23:03:23.453784+0000","last_clean":"2023-12-05T23:03:23.453784+0000","last_became_active":"2023-12-05T22:20:03.954510+0000","last_became_peered":"2023-12-05T22:20:03.954510+0000","last_unstale":"2023-12-05T23:03:23.453784+0000","last_undegraded":"2023-12-05T23:03:23.453784+0000","last_fullsized":"2023-12-05T23:03:23.453784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:23:12.821704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:58.459554+0000","last_change":"2023-12-05T22:20:03.955392+0000","last_active":"2023-12-05T23:03:58.459554+0000","last_peered":"2023-12-05T23:03:58.459554+0000","last_clean":"2023-12-05T23:03:58.459554+0000","last_became_active":"2023-12-05T22:20:03.954778+0000","last_became_peered":"2023-12-05T22:20:03.954778+0000","last_unstale":"2023-12-05T23:03:58.459554+0000","last_undegraded":"2023-12-05T23:03:58.459554+0000","last_fullsized":"2023-12-05T23:03:58.459554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:49:04.013308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458679+0000","last_change":"2023-12-05T22:20:03.954927+0000","last_active":"2023-12-05T23:03:53.458679+0000","last_peered":"2023-12-05T23:03:53.458679+0000","last_clean":"2023-12-05T23:03:53.458679+0000","last_became_active":"2023-12-05T22:20:03.954437+0000","last_became_peered":"2023-12-05T22:20:03.954437+0000","last_unstale":"2023-12-05T23:03:53.458679+0000","last_undegraded":"2023-12-05T23:03:53.458679+0000","last_fullsized":"2023-12-05T23:03:53.458679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T22:36:37.288695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.334539+0000","last_change":"2023-12-05T22:20:03.949404+0000","last_active":"2023-12-05T23:03:48.334539+0000","last_peered":"2023-12-05T23:03:48.334539+0000","last_clean":"2023-12-05T23:03:48.334539+0000","last_became_active":"2023-12-05T22:20:03.948733+0000","last_became_peered":"2023-12-05T22:20:03.948733+0000","last_unstale":"2023-12-05T23:03:48.334539+0000","last_undegraded":"2023-12-05T23:03:48.334539+0000","last_fullsized":"2023-12-05T23:03:48.334539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T01:35:07.902401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":489,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:48.399226+0000","last_change":"2023-12-05T22:20:03.950953+0000","last_active":"2023-12-05T23:03:48.399226+0000","last_peered":"2023-12-05T23:03:48.399226+0000","last_clean":"2023-12-05T23:03:48.399226+0000","last_became_active":"2023-12-05T22:20:03.950488+0000","last_became_peered":"2023-12-05T22:20:03.950488+0000","last_unstale":"2023-12-05T23:03:48.399226+0000","last_undegraded":"2023-12-05T23:03:48.399226+0000","last_fullsized":"2023-12-05T23:03:48.399226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:40:00.287834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:13.406015+0000","last_change":"2023-12-05T22:20:03.948835+0000","last_active":"2023-12-05T23:04:13.406015+0000","last_peered":"2023-12-05T23:04:13.406015+0000","last_clean":"2023-12-05T23:04:13.406015+0000","last_became_active":"2023-12-05T22:20:03.948291+0000","last_became_peered":"2023-12-05T22:20:03.948291+0000","last_unstale":"2023-12-05T23:04:13.406015+0000","last_undegraded":"2023-12-05T23:04:13.406015+0000","last_fullsized":"2023-12-05T23:04:13.406015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:00:52.805869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":488,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:03:53.458627+0000","last_change":"2023-12-05T22:20:03.944949+0000","last_active":"2023-12-05T23:03:53.458627+0000","last_peered":"2023-12-05T23:03:53.458627+0000","last_clean":"2023-12-05T23:03:53.458627+0000","last_became_active":"2023-12-05T22:20:03.944294+0000","last_became_peered":"2023-12-05T22:20:03.944294+0000","last_unstale":"2023-12-05T23:03:53.458627+0000","last_undegraded":"2023-12-05T23:03:53.458627+0000","last_fullsized":"2023-12-05T23:03:53.458627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:20:02.912112+0000","last_clean_scrub_stamp":"2023-12-05T22:20:02.912112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T00:52:50.330306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":619,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:04:08.461143+0000","last_change":"2023-12-05T22:20:00.475423+0000","last_active":"2023-12-05T23:04:08.461143+0000","last_peered":"2023-12-05T23:04:08.461143+0000","last_clean":"2023-12-05T23:04:08.461143+0000","last_became_active":"2023-12-05T22:20:00.473094+0000","last_became_peered":"2023-12-05T22:20:00.473094+0000","last_unstale":"2023-12-05T23:04:08.461143+0000","last_undegraded":"2023-12-05T23:04:08.461143+0000","last_fullsized":"2023-12-05T23:04:08.461143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T22:19:58.883597+0000","last_clean_scrub_stamp":"2023-12-05T22:19:58.883597+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:32:18.798306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147348,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94224492,"statfs":{"total":96636764160,"available":96485879808,"internally_reserved":0,"allocated":3874816,"data_stored":2074907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53700000000000003,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.21299999999999999},"max":{"1min":0.65500000000000003,"5min":0.86399999999999999,"15min":2.6389999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.621,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.23400000000000001},"max":{"1min":0.68000000000000005,"5min":0.81999999999999995,"15min":2.6779999999999999},"last":0.68999999999999995}]},{"osd":1,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.47399999999999998,"5min":0.46100000000000002,"15min":0.19},"max":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":1.3580000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67100000000000004,"15min":0.69599999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.26800000000000002},"max":{"1min":0.77300000000000002,"5min":0.90500000000000003,"15min":5.8959999999999999},"last":0.64600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738937,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121560,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250280,"statfs":{"total":96636764160,"available":96512286720,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56000000000000005,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.20300000000000001},"max":{"1min":0.73299999999999998,"5min":0.754,"15min":13.76},"last":0.625},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.249},"max":{"1min":0.66900000000000004,"5min":0.73399999999999999,"15min":8.4659999999999993},"last":0.54500000000000004}]},{"osd":2,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.622,"15min":0.66000000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.29599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":8.2270000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63500000000000001,"15min":0.69499999999999995},"min":{"1min":0.50900000000000001,"5min":0.48499999999999999,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":14.518000000000001},"last":0.74099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175704,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196136,"statfs":{"total":96636764160,"available":96456843264,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.214},"max":{"1min":0.69099999999999995,"5min":0.749,"15min":2.9390000000000001},"last":0.498},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.50900000000000001,"5min":0.495,"15min":0.23200000000000001},"max":{"1min":0.68100000000000005,"5min":0.78100000000000003,"15min":11.662000000000001},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.27100000000000002},"max":{"1min":0.79100000000000004,"5min":0.89600000000000002,"15min":2.714},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.29799999999999999},"max":{"1min":0.78400000000000003,"5min":0.96599999999999997,"15min":2.843},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:07:04.374 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-05T22:20:02.912112+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=20, tm_sec=2, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-05T22:19:58.883597+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=22, tm_min=19, tm_sec=58, tm_wday=1, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=5, tm_hour=23, tm_min=7, tm_sec=1, tm_wday=1, tm_yday=339, tm_isdst=0) 2023-12-05T23:07:04.375 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-05T23:07:24.376 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-05T23:07:24.676 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-12-05T23:07:24.676 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2023-12-05T23:07:24.693 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":1892,"stamp":"2023-12-05T23:07:23.348059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":444600,"kb_used_data":12512,"kb_used_omap":4,"kb_used_meta":432059,"kb_avail":282670920,"statfs":{"total":289910292480,"available":289455022080,"internally_reserved":0,"allocated":12812288,"data_stored":7405457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":442428751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003473"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":496,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:07.898596+0000","last_change":"2023-12-05T23:07:07.898617+0000","last_active":"2023-12-05T23:07:07.898596+0000","last_peered":"2023-12-05T23:07:07.898596+0000","last_clean":"2023-12-05T23:07:07.898596+0000","last_became_active":"2023-12-05T22:20:03.947069+0000","last_became_peered":"2023-12-05T22:20:03.947069+0000","last_unstale":"2023-12-05T23:07:07.898596+0000","last_undegraded":"2023-12-05T23:07:07.898596+0000","last_fullsized":"2023-12-05T23:07:07.898596+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:07.898480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:07.898480+0000","last_clean_scrub_stamp":"2023-12-05T23:07:07.898480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T05:58:25.258545+0000","scrub_duration":0.21160583399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":496,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:12.847907+0000","last_change":"2023-12-05T23:07:12.847959+0000","last_active":"2023-12-05T23:07:12.847907+0000","last_peered":"2023-12-05T23:07:12.847907+0000","last_clean":"2023-12-05T23:07:12.847907+0000","last_became_active":"2023-12-05T22:20:03.954510+0000","last_became_peered":"2023-12-05T22:20:03.954510+0000","last_unstale":"2023-12-05T23:07:12.847907+0000","last_undegraded":"2023-12-05T23:07:12.847907+0000","last_fullsized":"2023-12-05T23:07:12.847907+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:12.847668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:12.847668+0000","last_clean_scrub_stamp":"2023-12-05T23:07:12.847668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:26:41.779352+0000","scrub_duration":0.077862836000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":497,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:08.851639+0000","last_change":"2023-12-05T23:07:08.852587+0000","last_active":"2023-12-05T23:07:08.851639+0000","last_peered":"2023-12-05T23:07:08.851639+0000","last_clean":"2023-12-05T23:07:08.851639+0000","last_became_active":"2023-12-05T22:20:03.954778+0000","last_became_peered":"2023-12-05T22:20:03.954778+0000","last_unstale":"2023-12-05T23:07:08.851639+0000","last_undegraded":"2023-12-05T23:07:08.851639+0000","last_fullsized":"2023-12-05T23:07:08.851639+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:08.851405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:08.851405+0000","last_clean_scrub_stamp":"2023-12-05T23:07:08.851405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T09:55:20.451758+0000","scrub_duration":0.185320553,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":497,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:10.714313+0000","last_change":"2023-12-05T23:07:10.714361+0000","last_active":"2023-12-05T23:07:10.714313+0000","last_peered":"2023-12-05T23:07:10.714313+0000","last_clean":"2023-12-05T23:07:10.714313+0000","last_became_active":"2023-12-05T22:20:03.954437+0000","last_became_peered":"2023-12-05T22:20:03.954437+0000","last_unstale":"2023-12-05T23:07:10.714313+0000","last_undegraded":"2023-12-05T23:07:10.714313+0000","last_fullsized":"2023-12-05T23:07:10.714313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:10.714080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:10.714080+0000","last_clean_scrub_stamp":"2023-12-05T23:07:10.714080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T04:12:13.331511+0000","scrub_duration":0.0032977750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":498,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:14.008830+0000","last_change":"2023-12-05T23:07:14.008861+0000","last_active":"2023-12-05T23:07:14.008830+0000","last_peered":"2023-12-05T23:07:14.008830+0000","last_clean":"2023-12-05T23:07:14.008830+0000","last_became_active":"2023-12-05T22:20:03.948733+0000","last_became_peered":"2023-12-05T22:20:03.948733+0000","last_unstale":"2023-12-05T23:07:14.008830+0000","last_undegraded":"2023-12-05T23:07:14.008830+0000","last_fullsized":"2023-12-05T23:07:14.008830+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:14.008664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:14.008664+0000","last_clean_scrub_stamp":"2023-12-05T23:07:14.008664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:16:35.580907+0000","scrub_duration":0.079144518999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":498,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:04.657899+0000","last_change":"2023-12-05T23:07:04.657934+0000","last_active":"2023-12-05T23:07:04.657899+0000","last_peered":"2023-12-05T23:07:04.657899+0000","last_clean":"2023-12-05T23:07:04.657899+0000","last_became_active":"2023-12-05T22:20:03.950488+0000","last_became_peered":"2023-12-05T22:20:03.950488+0000","last_unstale":"2023-12-05T23:07:04.657899+0000","last_undegraded":"2023-12-05T23:07:04.657899+0000","last_fullsized":"2023-12-05T23:07:04.657899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:04.657738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:04.657738+0000","last_clean_scrub_stamp":"2023-12-05T23:07:04.657738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T03:41:00.909713+0000","scrub_duration":0.167420873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":499,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:06.623575+0000","last_change":"2023-12-05T23:07:06.623597+0000","last_active":"2023-12-05T23:07:06.623575+0000","last_peered":"2023-12-05T23:07:06.623575+0000","last_clean":"2023-12-05T23:07:06.623575+0000","last_became_active":"2023-12-05T22:20:03.948291+0000","last_became_peered":"2023-12-05T22:20:03.948291+0000","last_unstale":"2023-12-05T23:07:06.623575+0000","last_undegraded":"2023-12-05T23:07:06.623575+0000","last_fullsized":"2023-12-05T23:07:06.623575+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:06.623476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:06.623476+0000","last_clean_scrub_stamp":"2023-12-05T23:07:06.623476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T03:38:54.301676+0000","scrub_duration":0.13566587999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":497,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:06.903987+0000","last_change":"2023-12-05T23:07:06.904011+0000","last_active":"2023-12-05T23:07:06.903987+0000","last_peered":"2023-12-05T23:07:06.903987+0000","last_clean":"2023-12-05T23:07:06.903987+0000","last_became_active":"2023-12-05T22:20:03.944294+0000","last_became_peered":"2023-12-05T22:20:03.944294+0000","last_unstale":"2023-12-05T23:07:06.903987+0000","last_undegraded":"2023-12-05T23:07:06.903987+0000","last_fullsized":"2023-12-05T23:07:06.903987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-05T23:07:06.903867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-05T23:07:06.903867+0000","last_clean_scrub_stamp":"2023-12-05T23:07:06.903867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T02:39:29.005690+0000","scrub_duration":0.21529457299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"471'99","reported_seq":630,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-12-05T23:07:03.802950+0000","last_change":"2023-12-05T23:07:03.802978+0000","last_active":"2023-12-05T23:07:03.802950+0000","last_peered":"2023-12-05T23:07:03.802950+0000","last_clean":"2023-12-05T23:07:03.802950+0000","last_became_active":"2023-12-05T22:20:00.473094+0000","last_became_peered":"2023-12-05T22:20:00.473094+0000","last_unstale":"2023-12-05T23:07:03.802950+0000","last_undegraded":"2023-12-05T23:07:03.802950+0000","last_fullsized":"2023-12-05T23:07:03.802950+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'99","last_scrub_stamp":"2023-12-05T23:07:03.802875+0000","last_deep_scrub":"471'99","last_deep_scrub_stamp":"2023-12-05T23:07:03.802875+0000","last_clean_scrub_stamp":"2023-12-05T23:07:03.802875+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-07T10:33:06.313946+0000","scrub_duration":0.0063090830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738944,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147344,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94224496,"statfs":{"total":96636764160,"available":96485883904,"internally_reserved":0,"allocated":3874816,"data_stored":2074907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53700000000000003,"15min":0.54400000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.21299999999999999},"max":{"1min":0.65500000000000003,"5min":0.86399999999999999,"15min":2.6389999999999998},"last":0.62},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.621,"15min":0.64300000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.23400000000000001},"max":{"1min":0.68000000000000005,"5min":0.81999999999999995,"15min":2.6779999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Tue Dec 5 23:06:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.47399999999999998,"5min":0.46100000000000002,"15min":0.19},"max":{"1min":0.70599999999999996,"5min":0.76400000000000001,"15min":1.3580000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67100000000000004,"15min":0.69599999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.26800000000000002},"max":{"1min":0.77300000000000002,"5min":0.90500000000000003,"15min":5.8959999999999999},"last":0.71399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738941,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121556,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250284,"statfs":{"total":96636764160,"available":96512290816,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56000000000000005,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.20300000000000001},"max":{"1min":0.73299999999999998,"5min":0.754,"15min":13.76},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.249},"max":{"1min":0.66900000000000004,"5min":0.73399999999999999,"15min":8.4659999999999993},"last":0.499}]},{"osd":2,"last update":"Tue Dec 5 23:06:23 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.622,"15min":0.66000000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.29599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":8.2270000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63500000000000001,"15min":0.69499999999999995},"min":{"1min":0.50900000000000001,"5min":0.48499999999999999,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":14.518000000000001},"last":0.57299999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738943,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175700,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196140,"statfs":{"total":96636764160,"available":96456847360,"internally_reserved":0,"allocated":4468736,"data_stored":2665275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.214},"max":{"1min":0.69099999999999995,"5min":0.749,"15min":2.9390000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.50900000000000001,"5min":0.495,"15min":0.23200000000000001},"max":{"1min":0.68100000000000005,"5min":0.78100000000000003,"15min":11.662000000000001},"last":0.61299999999999999}]},{"osd":2,"last update":"Tue Dec 5 23:06:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.27100000000000002},"max":{"1min":0.79100000000000004,"5min":0.89600000000000002,"15min":2.714},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.29799999999999999},"max":{"1min":0.78400000000000003,"5min":0.96599999999999997,"15min":2.843},"last":0.65800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-05T23:07:24.695 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2023-12-05T23:07:25.007 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-05T23:07:25.008 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-05T23:07:25.008 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-05T23:07:25.008 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-12-05T23:07:25.009 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:07:25.104 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-12-05T23:07:25.104 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-12-05T23:07:25.104 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:07:25.195 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-12-05T23:07:25.195 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-12-05T23:07:25.195 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:07:25.271 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-12-05T23:07:25.271 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-05T23:07:25.271 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-12-05T23:07:25.271 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:07:25.298 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-12-05T23:07:25.298 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-05T23:07:25.298 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-12-05T23:07:25.298 INFO:teuthology.orchestra.run:waiting for 300 2023-12-05T23:07:25.356 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-12-05T23:07:25.356 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-05T23:07:25.356 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-05T23:07:25.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T23:07:25.408 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-12-05T23:07:25.694 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T23:07:25.695 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-12-05T23:07:26.039 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T23:07:26.040 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-12-05T23:07:26.213 INFO:tasks.ceph:Archiving mon data... 2023-12-05T23:07:26.213 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/data/cluster1.mon.a.tgz 2023-12-05T23:07:26.213 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2023-12-05T23:07:26.219 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.1XQuabWGpA 2023-12-05T23:07:26.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.1XQuabWGpA 2023-12-05T23:07:26.341 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.1XQuabWGpA 2023-12-05T23:07:26.407 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.1XQuabWGpA is 307KB 2023-12-05T23:07:26.464 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.1XQuabWGpA 2023-12-05T23:07:26.470 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-05T23:07:26.470 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-05T23:07:26.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-05T23:07:26.576 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-05T23:07:26.581 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-05T23:07:26.582 INFO:tasks.ceph:Archiving crash dumps... 2023-12-05T23:07:26.583 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/remote/smithi062/crash 2023-12-05T23:07:26.584 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-05T23:07:26.632 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/remote/smithi190/crash 2023-12-05T23:07:26.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-05T23:07:26.685 INFO:tasks.ceph:Compressing logs... 2023-12-05T23:07:26.686 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-05T23:07:26.688 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-05T23:07:26.712 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34814.log 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39815.log 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.34814.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34814.log.gz 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39741.log 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39815.log.gz 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39741.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39741.log.gz 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39919.log 2023-12-05T23:07:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41635.log 2023-12-05T23:07:26.716 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39659.log 2023-12-05T23:07:26.716 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39919.log.gz 2023-12-05T23:07:26.716 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34899.log 2023-12-05T23:07:26.716 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41635.log.gz 2023-12-05T23:07:26.716 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39659.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39659.log.gz 2023-12-05T23:07:26.716 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41713.log 2023-12-05T23:07:26.721 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.34899.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34899.log.gz 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41713.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31089.log 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41713.log.gz 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39477.log 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.tmp-client.admin.31089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41427.log 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31089.log.gz 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39477.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39477.log.gz 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39737.log 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41479.log 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41427.log: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39737.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41427.log.gz 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39737.log.gz 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41609.log 2023-12-05T23:07:26.722 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39667.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41479.log.gz 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41687.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41609.log.gz 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39824.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39667.log.gz 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34864.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41687.log.gz 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39824.log.gz 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.34864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41737.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34864.log.gz 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41661.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41453.log 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41737.log.gz 2023-12-05T23:07:26.723 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39426.log 2023-12-05T23:07:26.724 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41661.log.gz 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41453.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-mgr.x.log -- replaced with /var/log/ceph/cluster1-client.admin.41453.log.gz 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41531.log 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39426.log.gz 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39555.log 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41531.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41531.log.gz 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34840.log 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39555.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39555.log.gz 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.34840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39365.log 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34840.log.gz 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34967.log 2023-12-05T23:07:26.727 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39365.log: gzip -5 --verbose 0.0% -- /var/log/ceph/cluster1-client.admin.39745.log 2023-12-05T23:07:26.728 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.39365.log.gz 2023-12-05T23:07:26.728 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.34967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39529.log 2023-12-05T23:07:26.728 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34967.log.gz 2023-12-05T23:07:26.730 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39745.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39745.log.gz 2023-12-05T23:07:26.730 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41583.log 2023-12-05T23:07:26.730 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39819.log 2023-12-05T23:07:26.730 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39529.log.gz 2023-12-05T23:07:26.730 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41583.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39945.log 2023-12-05T23:07:26.730 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.41583.log.gz 2023-12-05T23:07:26.730 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39589.log 2023-12-05T23:07:26.731 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39819.log.gz/var/log/ceph/cluster1-client.admin.39945.log: 2023-12-05T23:07:26.731 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34720.log 2023-12-05T23:07:26.731 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39945.log.gz 2023-12-05T23:07:26.731 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39589.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39589.log.gz 2023-12-05T23:07:26.731 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41557.log 2023-12-05T23:07:26.731 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.34720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41505.log 2023-12-05T23:07:26.731 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34720.log.gz 2023-12-05T23:07:26.732 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41557.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41557.log.gz 2023-12-05T23:07:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39451.log 2023-12-05T23:07:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39503.log 2023-12-05T23:07:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.41505.log: 0.0%/var/log/ceph/cluster1-client.admin.39451.log: -- replaced with /var/log/ceph/cluster1-client.admin.41505.log.gz 2023-12-05T23:07:26.733 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39893.log 2023-12-05T23:07:26.733 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.39451.log.gz 2023-12-05T23:07:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39503.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39503.log.gz 2023-12-05T23:07:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-12-05T23:07:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39583.log 2023-12-05T23:07:26.734 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39893.log.gz 2023-12-05T23:07:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39588.log 2023-12-05T23:07:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-12-05T23:07:26.735 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39583.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39583.log.gz 2023-12-05T23:07:26.735 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-client.admin.39588.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39588.log.gz 2023-12-05T23:07:26.736 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39666.log 2023-12-05T23:07:26.736 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cluster1-osd.1.log: /var/log/ceph/cluster1-client.admin.39666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39666.log.gz 2023-12-05T23:07:26.739 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42607.log 2023-12-05T23:07:26.739 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58492.log 2023-12-05T23:07:26.739 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71415.log 2023-12-05T23:07:26.739 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42607.log: /var/log/ceph/cluster2-client.mirror.58492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53559.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71415.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71415.log.gz 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.61422.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42607.log.gz 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56344.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53559.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53559.log.gz 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.1.61422.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58492.log.gz 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66455.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79501.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66455.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56344.log.gz 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52325.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66455.log.gz 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78409.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79501.log.gz 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51178.log 2023-12-05T23:07:26.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53294.log 2023-12-05T23:07:26.741 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78409.log.gz 2023-12-05T23:07:26.741 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71685.log 2023-12-05T23:07:26.741 INFO:teuthology.orchestra.run.smithi190.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51178.log.gz 2023-12-05T23:07:26.741 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80671.log 2023-12-05T23:07:26.741 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53294.log.gz 2023-12-05T23:07:26.741 INFO:teuthology.orchestra.run.smithi062.stderr: 88.1% -- replaced with /var/log/ceph/cluster1.log.gz 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74681.log 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71685.log.gz 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80671.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80671.log.gz 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54831.log 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69930.log 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50630.log 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.69930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43135.log 2023-12-05T23:07:26.743 INFO:teuthology.orchestra.run.smithi190.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69930.log.gz 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54787.log 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43135.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50630.log.gz 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54831.log.gz 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62858.log 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65408.log 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43135.log.gz 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62858.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54787.log.gz 2023-12-05T23:07:26.744 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78877.log 2023-12-05T23:07:26.745 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.62858.log.gz 2023-12-05T23:07:26.745 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.75743.log 2023-12-05T23:07:26.745 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65408.log: /var/log/ceph/cluster2-client.admin.78877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65979.log 2023-12-05T23:07:26.746 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78877.log.gz 2023-12-05T23:07:26.746 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65408.log.gz 2023-12-05T23:07:26.746 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.1.75743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55864.log 2023-12-05T23:07:26.746 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57689.log 2023-12-05T23:07:26.746 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65979.log.gz 2023-12-05T23:07:26.746 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59350.log 2023-12-05T23:07:26.747 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65069.log 2023-12-05T23:07:26.747 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55864.log.gz 2023-12-05T23:07:26.747 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57689.log: /var/log/ceph/cluster2-client.mirror.59350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57842.log 2023-12-05T23:07:26.747 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65069.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.75743.log.gz 2023-12-05T23:07:26.747 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76283.log 2023-12-05T23:07:26.747 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57689.log.gz 2023-12-05T23:07:26.747 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59350.log.gz 2023-12-05T23:07:26.748 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68401.log 2023-12-05T23:07:26.749 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68484.log 2023-12-05T23:07:26.749 INFO:teuthology.orchestra.run.smithi190.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76283.log.gz 2023-12-05T23:07:26.749 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68401.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65069.log.gz 2023-12-05T23:07:26.749 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67781.log 2023-12-05T23:07:26.750 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52283.log 2023-12-05T23:07:26.750 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57842.log.gz 2023-12-05T23:07:26.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68484.log: /var/log/ceph/cluster2-client.mirror.67781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45840.log 2023-12-05T23:07:26.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50452.log 2023-12-05T23:07:26.751 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5%/var/log/ceph/cluster2-client.mirror.45840.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68484.log.gz 2023-12-05T23:07:26.751 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52283.log.gz 2023-12-05T23:07:26.751 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68401.log.gz 2023-12-05T23:07:26.751 INFO:teuthology.orchestra.run.smithi190.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67781.log.gz 2023-12-05T23:07:26.752 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.76522.log 2023-12-05T23:07:26.752 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68629.log 2023-12-05T23:07:26.753 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.76522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56154.log 2023-12-05T23:07:26.753 INFO:teuthology.orchestra.run.smithi190.stderr:gzip/var/log/ceph/cluster2-client.mirror.68629.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45730.log 2023-12-05T23:07:26.753 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50452.log.gz 2023-12-05T23:07:26.754 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68629.log.gz 2023-12-05T23:07:26.754 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56154.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.76522.log.gz 2023-12-05T23:07:26.754 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56154.log.gz 2023-12-05T23:07:26.754 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61644.log 2023-12-05T23:07:26.755 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71775.log 2023-12-05T23:07:26.755 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61644.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45730.log.gz 2023-12-05T23:07:26.755 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46309.log 2023-12-05T23:07:26.756 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61644.log.gz 2023-12-05T23:07:26.756 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75370.log 2023-12-05T23:07:26.757 INFO:teuthology.orchestra.run.smithi190.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71775.log.gz 2023-12-05T23:07:26.757 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45840.log.gz 2023-12-05T23:07:26.757 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54307.log 2023-12-05T23:07:26.757 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69071.log 2023-12-05T23:07:26.757 INFO:teuthology.orchestra.run.smithi190.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75370.log.gz/var/log/ceph/cluster1-client.mirror.54307.log: 2023-12-05T23:07:26.757 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68257.log 2023-12-05T23:07:26.758 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54307.log.gz 2023-12-05T23:07:26.758 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49931.log 2023-12-05T23:07:26.758 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39596.log 2023-12-05T23:07:26.759 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5%/var/log/ceph/cluster2-client.mirror.49931.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68257.log.gz 2023-12-05T23:07:26.759 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69071.log.gz 2023-12-05T23:07:26.759 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61767.log 2023-12-05T23:07:26.765 INFO:teuthology.orchestra.run.smithi190.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49931.log.gz 2023-12-05T23:07:26.765 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39596.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39596.log.gz 2023-12-05T23:07:26.765 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63513.log 2023-12-05T23:07:26.765 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55221.log 2023-12-05T23:07:26.765 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.63513.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61767.log.gz 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60064.log 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55221.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63513.log.gz 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40687.log 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55221.log.gz 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46245.log 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40687.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46309.log.gz 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.60064.log.gz -- 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr: /var/log/ceph/cluster2-client.mirror.66302.log 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40687.log.gz 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76320.log 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66302.log: gzip 89.1% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.46245.log.gz --verbose 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr: -- /var/log/ceph/cluster2-client.mirror.56000.log 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46018.log 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56000.log: 89.8% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66302.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.76320.log.gz 2023-12-05T23:07:26.766 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54478.log 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68670.log 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75833.log 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46018.log.gz/var/log/ceph/cluster2-client.mirror.68670.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54478.log.gz 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82570.log 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74702.log 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56000.log.gz 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75833.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68670.log.gz 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82570.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82570.log.gz 2023-12-05T23:07:26.767 INFO:teuthology.orchestra.run.smithi190.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75833.log.gz 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69278.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68215.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74702.log.gz 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52266.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62922.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69278.log.gz 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57321.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52266.log.gz 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77316.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82856.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3%/var/log/ceph/cluster2-client.admin.77316.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57321.log.gz 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76656.log 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77316.log.gz 2023-12-05T23:07:26.771 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82856.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68215.log.gz 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82856.log.gz 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62922.log.gz 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76364.log 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76656.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45097.log 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.76364.log: -- replaced with /var/log/ceph/cluster2-client.mirror.76656.log.gz 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53257.log 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr: 80.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76364.log.gz 2023-12-05T23:07:26.772 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83012.log 2023-12-05T23:07:26.775 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45097.log: /var/log/ceph/cluster2-client.mirror.53257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49968.log 2023-12-05T23:07:26.775 INFO:teuthology.orchestra.run.smithi190.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45097.log.gz 2023-12-05T23:07:26.775 INFO:teuthology.orchestra.run.smithi190.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53257.log.gz 2023-12-05T23:07:26.775 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80021.log 2023-12-05T23:07:26.775 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83012.log.gz 2023-12-05T23:07:26.775 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46860.log 2023-12-05T23:07:26.775 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49968.log: /var/log/ceph/cluster2-client.admin.80021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45143.log 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80021.log.gz 87.5% 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49968.log.gz 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73603.log 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47254.log 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73603.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45143.log.gz 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44491.log 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47254.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73603.log.gz 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73353.log 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45942.log 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73353.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44491.log.gz 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62106.log 2023-12-05T23:07:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59233.log 2023-12-05T23:07:26.779 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.45942.log: /var/log/ceph/cluster1-client.mirror.62106.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.45942.log.gz 2023-12-05T23:07:26.779 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62106.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59489.log 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75743.log 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73353.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59233.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59489.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59489.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46860.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53805.log 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78929.log 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75743.log: 74.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75743.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82700.log 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53805.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53386.log 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78929.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47254.log.gz 2023-12-05T23:07:26.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71077.log 2023-12-05T23:07:26.781 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82700.log.gz 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51380.log 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48080.log 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71077.log.gz 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56557.log 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53386.log.gz 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51380.log.gz 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53618.log 2023-12-05T23:07:26.784 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65028.log 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56557.log: /var/log/ceph/cluster2-client.mirror.53618.log: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79553.log 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48080.log.gz 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48140.log 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.68795.log 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65028.log.gz 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79553.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79553.log.gz 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69997.log 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68711.log 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48140.log.gz 2023-12-05T23:07:26.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.68795.log: /var/log/ceph/cluster1-client.mirror.69997.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69997.log.gz 2023-12-05T23:07:26.787 INFO:teuthology.orchestra.run.smithi190.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.68795.log.gz 2023-12-05T23:07:26.788 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71865.log 2023-12-05T23:07:26.788 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.50155.log 2023-12-05T23:07:26.789 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67368.log 2023-12-05T23:07:26.789 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71865.log.gz 2023-12-05T23:07:26.789 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.0.50155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46948.log 2023-12-05T23:07:26.791 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67368.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68711.log.gz 2023-12-05T23:07:26.791 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67368.log.gz 2023-12-05T23:07:26.791 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50472.log 2023-12-05T23:07:26.791 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74832.log 2023-12-05T23:07:26.793 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7%/var/log/ceph/cluster2-client.mirror.50472.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46948.log.gz 2023-12-05T23:07:26.793 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50472.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47360.log 2023-12-05T23:07:26.793 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.793 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51020.log 2023-12-05T23:07:26.794 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74832.log.gz 2023-12-05T23:07:26.794 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68133.log 2023-12-05T23:07:26.794 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51020.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51020.log.gz 2023-12-05T23:07:26.795 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39514.log 2023-12-05T23:07:26.796 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68133.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68133.log.gz 2023-12-05T23:07:26.796 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43398.log 2023-12-05T23:07:26.797 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39514.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39514.log.gz 2023-12-05T23:07:26.797 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78304.log 2023-12-05T23:07:26.798 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43398.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.61422.log.gz 2023-12-05T23:07:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51195.log 2023-12-05T23:07:26.799 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78304.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78304.log.gz 2023-12-05T23:07:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79371.log 2023-12-05T23:07:26.800 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51195.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47360.log.gz 2023-12-05T23:07:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55072.log 2023-12-05T23:07:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79371.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79371.log.gz 2023-12-05T23:07:26.802 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51195.log.gz 2023-12-05T23:07:26.802 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81269.log 2023-12-05T23:07:26.803 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.40238.log 2023-12-05T23:07:26.803 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55072.log: /var/log/ceph/cluster2-client.admin.81269.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81269.log.gz 2023-12-05T23:07:26.803 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50800.log 2023-12-05T23:07:26.804 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.1.40238.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55072.log.gz 2023-12-05T23:07:26.804 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43376.log 2023-12-05T23:07:26.806 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50800.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50800.log.gz 2023-12-05T23:07:26.806 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63559.log 2023-12-05T23:07:26.806 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5%/var/log/ceph/cluster2-client.mirror.43376.log: -- replaced with /var/log/ceph/cluster1-client.mirror.1.40238.log.gz 2023-12-05T23:07:26.807 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69298.log 2023-12-05T23:07:26.807 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63559.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43376.log.gz 2023-12-05T23:07:26.808 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61285.log 2023-12-05T23:07:26.808 INFO:teuthology.orchestra.run.smithi190.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63559.log.gz 2023-12-05T23:07:26.808 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75491.log 2023-12-05T23:07:26.810 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61285.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61285.log.gz 2023-12-05T23:07:26.810 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82596.log 2023-12-05T23:07:26.811 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75491.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69298.log.gz 2023-12-05T23:07:26.811 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75491.log.gz 2023-12-05T23:07:26.811 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74516.log 2023-12-05T23:07:26.813 INFO:teuthology.orchestra.run.smithi190.stderr: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63580.log 2023-12-05T23:07:26.813 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56557.log.gz 2023-12-05T23:07:26.813 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82596.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82596.log.gz 2023-12-05T23:07:26.813 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74516.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74516.log.gz 2023-12-05T23:07:26.813 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52187.log 2023-12-05T23:07:26.813 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79449.log 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63580.log: /var/log/ceph/cluster2-client.mirror.52187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83142.log 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52187.log.gz 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79449.log.gz 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74852.log 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46776.log 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83142.log.gz 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50261.log 2023-12-05T23:07:26.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46776.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63580.log.gz 2023-12-05T23:07:26.817 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68071.log 2023-12-05T23:07:26.817 INFO:teuthology.orchestra.run.smithi190.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46776.log.gz 2023-12-05T23:07:26.817 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82387.log 2023-12-05T23:07:26.817 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68071.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74852.log.gz 2023-12-05T23:07:26.817 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50261.log.gz 2023-12-05T23:07:26.817 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68071.log.gz 2023-12-05T23:07:26.817 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74539.log 2023-12-05T23:07:26.819 INFO:teuthology.orchestra.run.smithi190.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.50155.log.gz 2023-12-05T23:07:26.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57164.log 2023-12-05T23:07:26.819 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82387.log.gz 2023-12-05T23:07:26.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.76364.log 2023-12-05T23:07:26.819 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74539.log.gz 2023-12-05T23:07:26.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54656.log 2023-12-05T23:07:26.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.1.76364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49235.log 2023-12-05T23:07:26.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54656.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81139.log 2023-12-05T23:07:26.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49235.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54656.log.gz 2023-12-05T23:07:26.820 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57164.log.gz 2023-12-05T23:07:26.824 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55780.log 2023-12-05T23:07:26.824 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74560.log 2023-12-05T23:07:26.825 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81139.log.gz 2023-12-05T23:07:26.825 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69154.log 2023-12-05T23:07:26.826 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55780.log: /var/log/ceph/cluster2-client.mirror.74560.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74560.log.gz 2023-12-05T23:07:26.826 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45860.log 2023-12-05T23:07:26.827 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69154.log: 88.4% 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55780.log.gz 2023-12-05T23:07:26.827 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69154.log.gz 2023-12-05T23:07:26.827 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52167.log 2023-12-05T23:07:26.828 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80645.log 2023-12-05T23:07:26.834 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45860.log: /var/log/ceph/cluster2-client.mirror.52167.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45860.log.gz 2023-12-05T23:07:26.834 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47403.log 2023-12-05T23:07:26.835 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80645.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80645.log.gz 2023-12-05T23:07:26.835 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74725.log 2023-12-05T23:07:26.841 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47403.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47403.log.gz 2023-12-05T23:07:26.842 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54741.log 2023-12-05T23:07:26.842 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74725.log: 95.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.76364.log.gz 2023-12-05T23:07:26.843 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74725.log.gz 2023-12-05T23:07:26.844 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70672.log 2023-12-05T23:07:26.844 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42464.log 2023-12-05T23:07:26.844 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54741.log.gz 2023-12-05T23:07:26.844 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70672.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70672.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71370.log 2023-12-05T23:07:26.844 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.42464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72114.log 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.42464.log.gz 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71370.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71370.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76522.log 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.72114.log: 3.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72114.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76775.log 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70020.log 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76775.log: 73.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76522.log.gz 2023-12-05T23:07:26.847 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76775.log.gz 2023-12-05T23:07:26.848 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71617.log 2023-12-05T23:07:26.849 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77524.log 2023-12-05T23:07:26.849 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70020.log: /var/log/ceph/cluster1-client.mirror.71617.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70020.log.gz 2023-12-05T23:07:26.849 INFO:teuthology.orchestra.run.smithi190.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71617.log.gz 2023-12-05T23:07:26.849 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79969.log 2023-12-05T23:07:26.849 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83272.log 2023-12-05T23:07:26.850 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77524.log: /var/log/ceph/cluster2-client.admin.79969.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77524.log.gz 2023-12-05T23:07:26.850 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79969.log.gz 2023-12-05T23:07:26.850 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54218.log 2023-12-05T23:07:26.850 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47145.log 2023-12-05T23:07:26.851 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83272.log: /var/log/ceph/cluster2-client.mirror.54218.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83272.log.gz 2023-12-05T23:07:26.851 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80775.log 2023-12-05T23:07:26.851 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47145.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54218.log.gz 2023-12-05T23:07:26.857 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47145.log.gz 2023-12-05T23:07:26.857 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66478.log 2023-12-05T23:07:26.857 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69340.log 2023-12-05T23:07:26.858 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66478.log: /var/log/ceph/cluster2-client.admin.80775.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80775.log.gz 2023-12-05T23:07:26.858 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66478.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81009.log 2023-12-05T23:07:26.858 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.860 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57298.log 2023-12-05T23:07:26.860 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81009.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81009.log.gz 2023-12-05T23:07:26.860 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69340.log.gz 2023-12-05T23:07:26.860 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58275.log 2023-12-05T23:07:26.860 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61667.log 2023-12-05T23:07:26.861 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57298.log.gz/var/log/ceph/cluster2-client.mirror.58275.log: 2023-12-05T23:07:26.861 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67305.log 2023-12-05T23:07:26.861 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61667.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61667.log.gz 2023-12-05T23:07:26.862 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55094.log 2023-12-05T23:07:26.862 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67305.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58275.log.gz 2023-12-05T23:07:26.863 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57822.log 2023-12-05T23:07:26.863 INFO:teuthology.orchestra.run.smithi190.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67305.log.gz/var/log/ceph/cluster2-client.mirror.55094.log: 2023-12-05T23:07:26.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41097.log 2023-12-05T23:07:26.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57822.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55094.log.gz 2023-12-05T23:07:26.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53851.log 2023-12-05T23:07:26.865 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41097.log: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41097.log.gz 2023-12-05T23:07:26.866 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78721.log 2023-12-05T23:07:26.866 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57822.log.gz 2023-12-05T23:07:26.866 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77498.log 2023-12-05T23:07:26.867 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78721.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78721.log.gz 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53851.log.gz 2023-12-05T23:07:26.867 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.867 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68298.log 2023-12-05T23:07:26.867 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56088.log 2023-12-05T23:07:26.868 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77498.log: /var/log/ceph/cluster2-client.mirror.68298.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77498.log.gz 2023-12-05T23:07:26.868 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62040.log 2023-12-05T23:07:26.868 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68298.log.gz/var/log/ceph/cluster2-client.mirror.56088.log: 2023-12-05T23:07:26.869 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56088.log.gz 2023-12-05T23:07:26.869 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67389.log 2023-12-05T23:07:26.870 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51136.log 2023-12-05T23:07:26.870 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62040.log: /var/log/ceph/cluster2-client.mirror.67389.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67389.log.gz 2023-12-05T23:07:26.871 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62040.log.gz 2023-12-05T23:07:26.871 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40353.log 2023-12-05T23:07:26.872 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43334.log 2023-12-05T23:07:26.872 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51136.log: /var/log/ceph/cluster1-client.mirror.40353.log: 81.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40353.log.gz 2023-12-05T23:07:26.872 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51136.log.gz 2023-12-05T23:07:26.872 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67327.log 2023-12-05T23:07:26.873 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80515.log 2023-12-05T23:07:26.873 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43334.log: /var/log/ceph/cluster2-client.mirror.67327.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67327.log.gz 2023-12-05T23:07:26.878 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43334.log.gz 2023-12-05T23:07:26.879 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31211.log 2023-12-05T23:07:26.880 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78200.log 2023-12-05T23:07:26.880 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80515.log: /var/log/ceph/ceph.tmp-client.admin.31211.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80515.log.gz 2023-12-05T23:07:26.880 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31211.log.gz 2023-12-05T23:07:26.880 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81373.log 2023-12-05T23:07:26.881 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52345.log 2023-12-05T23:07:26.881 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78200.log: /var/log/ceph/cluster2-client.admin.81373.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78200.log.gz 2023-12-05T23:07:26.881 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81373.log.gz 2023-12-05T23:07:26.881 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40235.log 2023-12-05T23:07:26.882 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79085.log 2023-12-05T23:07:26.882 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52345.log: /var/log/ceph/cluster2-client.mirror.40235.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52345.log.gz 2023-12-05T23:07:26.882 INFO:teuthology.orchestra.run.smithi190.stderr: 82.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40235.log.gz 2023-12-05T23:07:26.882 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62323.log 2023-12-05T23:07:26.883 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65498.log 2023-12-05T23:07:26.884 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79085.log: /var/log/ceph/cluster1-client.mirror.62323.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79085.log.gz 2023-12-05T23:07:26.884 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40721.log 2023-12-05T23:07:26.884 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62323.log.gz 2023-12-05T23:07:26.885 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65498.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65498.log.gz 2023-12-05T23:07:26.886 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61374.log 2023-12-05T23:07:26.886 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82830.log 2023-12-05T23:07:26.886 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40721.log: /var/log/ceph/cluster1-client.mirror.61374.log: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40721.log.gz 2023-12-05T23:07:26.887 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61374.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78513.log 2023-12-05T23:07:26.887 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:26.887 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82830.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82830.log.gz 2023-12-05T23:07:26.888 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46396.log 2023-12-05T23:07:26.888 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52009.log 2023-12-05T23:07:26.888 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78513.log: /var/log/ceph/cluster2-client.mirror.46396.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78513.log.gz 2023-12-05T23:07:26.889 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59556.log 2023-12-05T23:07:26.892 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52009.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52009.log.gz 2023-12-05T23:07:26.893 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53539.log 2023-12-05T23:07:26.895 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59556.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59556.log.gz 2023-12-05T23:07:26.895 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71527.log 2023-12-05T23:07:26.912 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58832.log 2023-12-05T23:07:26.913 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71527.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71527.log.gz 2023-12-05T23:07:26.932 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43791.log 2023-12-05T23:07:26.933 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58832.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58832.log.gz 2023-12-05T23:07:26.940 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39677.log 2023-12-05T23:07:26.948 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81789.log 2023-12-05T23:07:26.949 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39677.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39677.log.gz 2023-12-05T23:07:26.964 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57621.log 2023-12-05T23:07:26.965 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81789.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81789.log.gz 2023-12-05T23:07:26.975 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77108.log 2023-12-05T23:07:26.990 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57621.log: 92.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52667.log 2023-12-05T23:07:26.990 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77108.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77108.log.gz 2023-12-05T23:07:26.993 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57621.log.gz 2023-12-05T23:07:26.993 INFO:teuthology.orchestra.run.smithi062.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-12-05T23:07:27.008 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58055.log 2023-12-05T23:07:27.009 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52667.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52667.log.gz 2023-12-05T23:07:27.032 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49255.log 2023-12-05T23:07:27.034 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58055.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58055.log.gz 2023-12-05T23:07:27.044 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53961.log 2023-12-05T23:07:27.045 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49255.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49255.log.gz 2023-12-05T23:07:27.054 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43398.log.gz 2023-12-05T23:07:27.060 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43224.log 2023-12-05T23:07:27.062 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53961.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53961.log.gz 2023-12-05T23:07:27.063 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74681.log.gz 2023-12-05T23:07:27.072 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57755.log 2023-12-05T23:07:27.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43224.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43224.log.gz 2023-12-05T23:07:27.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60630.log 2023-12-05T23:07:27.074 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57755.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57755.log.gz 2023-12-05T23:07:27.075 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47211.log 2023-12-05T23:07:27.075 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9%/var/log/ceph/cluster2-client.mirror.60630.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53618.log.gz 2023-12-05T23:07:27.076 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60630.log.gz 2023-12-05T23:07:27.076 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64936.log 2023-12-05T23:07:27.077 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66622.log 2023-12-05T23:07:27.077 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47211.log: /var/log/ceph/cluster1-client.mirror.64936.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64936.log.gz 2023-12-05T23:07:27.082 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47211.log.gz 2023-12-05T23:07:27.083 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64698.log 2023-12-05T23:07:27.083 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61811.log 2023-12-05T23:07:27.084 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66622.log: /var/log/ceph/cluster2-client.mirror.64698.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66622.log.gz 2023-12-05T23:07:27.084 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49869.log 2023-12-05T23:07:27.087 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61811.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61811.log.gz 2023-12-05T23:07:27.087 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49336.log 2023-12-05T23:07:27.088 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49869.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49869.log.gz 2023-12-05T23:07:27.089 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75556.log 2023-12-05T23:07:27.093 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49336.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49336.log.gz 2023-12-05T23:07:27.093 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48331.log 2023-12-05T23:07:27.093 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75556.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75556.log.gz 2023-12-05T23:07:27.094 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.40239.log 2023-12-05T23:07:27.095 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48331.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64698.log.gz 2023-12-05T23:07:27.095 INFO:teuthology.orchestra.run.smithi190.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48331.log.gz 2023-12-05T23:07:27.095 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71910.log 2023-12-05T23:07:27.095 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83298.log 2023-12-05T23:07:27.096 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.2.40239.log: /var/log/ceph/cluster1-client.mirror.71910.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71910.log.gz 2023-12-05T23:07:27.106 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49235.log.gz 2023-12-05T23:07:27.109 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70132.log 2023-12-05T23:07:27.109 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56910.log 2023-12-05T23:07:27.109 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83298.log.gz 2023-12-05T23:07:27.111 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70132.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70132.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64959.log 2023-12-05T23:07:27.111 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:27.112 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66768.log 2023-12-05T23:07:27.112 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64959.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56910.log.gz 2023-12-05T23:07:27.112 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64959.log.gz 2023-12-05T23:07:27.112 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62238.log 2023-12-05T23:07:27.112 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56200.log 2023-12-05T23:07:27.112 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62238.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66768.log.gz 2023-12-05T23:07:27.113 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76041.log 2023-12-05T23:07:27.113 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56200.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62238.log.gz 2023-12-05T23:07:27.113 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58162.log 2023-12-05T23:07:27.113 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56200.log.gz 2023-12-05T23:07:27.114 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45525.log 2023-12-05T23:07:27.114 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58162.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76041.log.gz 2023-12-05T23:07:27.114 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58162.log.gz 2023-12-05T23:07:27.114 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65129.log 2023-12-05T23:07:27.115 INFO:teuthology.orchestra.run.smithi062.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-12-05T23:07:27.116 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47340.log 2023-12-05T23:07:27.116 INFO:teuthology.orchestra.run.smithi190.stderr: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45525.log.gz 2023-12-05T23:07:27.116 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51866.log 2023-12-05T23:07:27.116 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47340.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65129.log.gz 2023-12-05T23:07:27.116 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78174.log 2023-12-05T23:07:27.117 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51866.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51866.log.gz 2023-12-05T23:07:27.118 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44515.log 2023-12-05T23:07:27.118 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78174.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78174.log.gz 2023-12-05T23:07:27.119 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60041.log 2023-12-05T23:07:27.120 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44515.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44515.log.gz 2023-12-05T23:07:27.121 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.74426.log 2023-12-05T23:07:27.121 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60041.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47340.log.gz 2023-12-05T23:07:27.121 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60041.log.gz 2023-12-05T23:07:27.122 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40520.log 2023-12-05T23:07:27.122 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.2.74426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45340.log 2023-12-05T23:07:27.123 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40520.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40520.log.gz 2023-12-05T23:07:27.123 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77472.log 2023-12-05T23:07:27.131 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45340.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.74426.log.gz 2023-12-05T23:07:27.132 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73733.log 2023-12-05T23:07:27.133 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77472.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77472.log.gz 2023-12-05T23:07:27.133 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53539.log.gz 2023-12-05T23:07:27.133 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48831.log 2023-12-05T23:07:27.134 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73733.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73733.log.gz 2023-12-05T23:07:27.135 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80619.log 2023-12-05T23:07:27.135 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9%/var/log/ceph/cluster2-client.mirror.48831.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52325.log.gz 2023-12-05T23:07:27.136 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61422.log 2023-12-05T23:07:27.136 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54939.log 2023-12-05T23:07:27.136 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61422.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80619.log.gz 2023-12-05T23:07:27.136 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55887.log 2023-12-05T23:07:27.136 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54939.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54939.log.gz 2023-12-05T23:07:27.137 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68546.log 2023-12-05T23:07:27.137 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55887.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55887.log.gz 2023-12-05T23:07:27.138 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71347.log 2023-12-05T23:07:27.139 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68546.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68546.log.gz 2023-12-05T23:07:27.139 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51116.log 2023-12-05T23:07:27.140 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71347.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71347.log.gz 2023-12-05T23:07:27.140 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76454.log 2023-12-05T23:07:27.141 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51116.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51116.log.gz 2023-12-05T23:07:27.142 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51768.log 2023-12-05T23:07:27.142 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76454.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76454.log.gz 2023-12-05T23:07:27.143 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61130.log 2023-12-05T23:07:27.144 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51768.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51768.log.gz 2023-12-05T23:07:27.144 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.40235.log 2023-12-05T23:07:27.145 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61130.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61130.log.gz 2023-12-05T23:07:27.145 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53100.log 2023-12-05T23:07:27.150 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.0.40235.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.40235.log.gz 2023-12-05T23:07:27.151 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56263.log 2023-12-05T23:07:27.151 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46396.log.gz 2023-12-05T23:07:27.152 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53100.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53100.log.gz 2023-12-05T23:07:27.152 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73955.log 2023-12-05T23:07:27.153 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70582.log 2023-12-05T23:07:27.153 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56263.log: /var/log/ceph/cluster1-client.mirror.73955.log: 79.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73955.log.gz 2023-12-05T23:07:27.154 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56263.log.gz 2023-12-05T23:07:27.154 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62346.log 2023-12-05T23:07:27.155 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72022.log 2023-12-05T23:07:27.155 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70582.log: /var/log/ceph/cluster1-client.mirror.62346.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70582.log.gz 2023-12-05T23:07:27.155 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62346.log.gz 2023-12-05T23:07:27.155 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65005.log 2023-12-05T23:07:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67202.log 2023-12-05T23:07:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.72022.log: /var/log/ceph/cluster1-client.mirror.65005.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72022.log.gz 2023-12-05T23:07:27.157 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65005.log.gz 2023-12-05T23:07:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52625.log 2023-12-05T23:07:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67202.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67202.log.gz 2023-12-05T23:07:27.158 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70537.log 2023-12-05T23:07:27.158 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79163.log 2023-12-05T23:07:27.159 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52625.log: /var/log/ceph/cluster1-client.mirror.70537.log: 86.8% 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52625.log.gz 2023-12-05T23:07:27.159 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70537.log.gz 2023-12-05T23:07:27.160 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67513.log 2023-12-05T23:07:27.160 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69712.log 2023-12-05T23:07:27.160 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79163.log: /var/log/ceph/cluster2-client.mirror.67513.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79163.log.gz 2023-12-05T23:07:27.160 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63098.log 2023-12-05T23:07:27.161 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67513.log.gz 2023-12-05T23:07:27.161 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69712.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69712.log.gz 2023-12-05T23:07:27.161 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52382.log 2023-12-05T23:07:27.162 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79605.log 2023-12-05T23:07:27.162 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63098.log: /var/log/ceph/cluster2-client.mirror.52382.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63098.log.gz 2023-12-05T23:07:27.162 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52382.log.gz 2023-12-05T23:07:27.162 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51057.log 2023-12-05T23:07:27.164 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69588.log 2023-12-05T23:07:27.164 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79605.log: /var/log/ceph/cluster2-client.mirror.51057.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79605.log.gz 2023-12-05T23:07:27.164 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40704.log 2023-12-05T23:07:27.164 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51057.log.gz 2023-12-05T23:07:27.164 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69588.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61422.log.gz 2023-12-05T23:07:27.166 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56798.log 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69588.log.gz 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49889.log 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr: 56.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40704.log.gz/var/log/ceph/cluster2-client.mirror.56798.log: 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68753.log 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71325.log 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49889.log: 89.7%/var/log/ceph/cluster2-client.mirror.68753.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49889.log.gz 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56798.log.gz 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54718.log 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68753.log.gz/var/log/ceph/cluster1-client.mirror.71325.log: 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53164.log 2023-12-05T23:07:27.167 INFO:teuthology.orchestra.run.smithi190.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71325.log.gz 2023-12-05T23:07:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54718.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54718.log.gz 2023-12-05T23:07:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78070.log 2023-12-05T23:07:27.169 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70335.log 2023-12-05T23:07:27.169 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71122.log 2023-12-05T23:07:27.169 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78070.log.gz 2023-12-05T23:07:27.169 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70335.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70335.log.gz 2023-12-05T23:07:27.170 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57991.log 2023-12-05T23:07:27.170 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43727.log 2023-12-05T23:07:27.175 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71122.log: /var/log/ceph/cluster1-client.mirror.57991.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71122.log.gz 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr: 95.5% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57991.log.gz 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53164.log.gz 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40206.log 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51475.log 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43727.log: /var/log/ceph/cluster2-client.admin.40206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70222.log 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40206.log.gz 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51475.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51475.log.gz 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55802.log 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69732.log 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70222.log.gz 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55802.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55802.log.gz 2023-12-05T23:07:27.176 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70065.log 2023-12-05T23:07:27.177 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71167.log 2023-12-05T23:07:27.177 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70065.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70065.log.gz 2023-12-05T23:07:27.177 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79345.log 2023-12-05T23:07:27.177 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71167.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71167.log.gz 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60285.log 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43727.log.gz 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79345.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79345.log.gz 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69732.log.gz 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46753.log 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44760.log 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46753.log: gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/cluster1-client.mirror.60285.log.gz /var/log/ceph/cluster1-client.mirror.73203.log 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:27.179 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44760.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46753.log.gz 2023-12-05T23:07:27.180 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46133.log 2023-12-05T23:07:27.180 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81659.log 2023-12-05T23:07:27.181 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73203.log: /var/log/ceph/cluster2-client.mirror.46133.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73203.log.gz 2023-12-05T23:07:27.181 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51274.log 2023-12-05T23:07:27.182 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81659.log: 90.0% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81659.log.gz 2023-12-05T23:07:27.182 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46133.log.gz 2023-12-05T23:07:27.182 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44760.log.gz 2023-12-05T23:07:27.183 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60239.log 2023-12-05T23:07:27.183 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77134.log 2023-12-05T23:07:27.184 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59400.log 2023-12-05T23:07:27.184 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77134.log: 89.7% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60239.log.gz 2023-12-05T23:07:27.184 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77134.log.gz 2023-12-05T23:07:27.185 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51274.log.gz 2023-12-05T23:07:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46503.log 2023-12-05T23:07:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59256.log 2023-12-05T23:07:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63467.log 2023-12-05T23:07:27.186 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59256.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59400.log.gz 2023-12-05T23:07:27.187 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59256.log.gz 2023-12-05T23:07:27.187 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60566.log 2023-12-05T23:07:27.187 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63254.log 2023-12-05T23:07:27.187 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60566.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63467.log.gz 91.8% 2023-12-05T23:07:27.187 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46503.log.gz 2023-12-05T23:07:27.187 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60566.log.gz 2023-12-05T23:07:27.188 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-12-05T23:07:27.188 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66706.log 2023-12-05T23:07:27.188 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-osd.1.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63254.log.gz 2023-12-05T23:07:27.188 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48656.log 2023-12-05T23:07:27.189 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75259.log 2023-12-05T23:07:27.190 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.48656.log: /var/log/ceph/cluster2-client.mirror.66706.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66706.log.gz 2023-12-05T23:07:27.190 INFO:teuthology.orchestra.run.smithi190.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48656.log.gz 2023-12-05T23:07:27.190 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68091.log 2023-12-05T23:07:27.190 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68649.log 2023-12-05T23:07:27.191 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68091.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75259.log.gz 2023-12-05T23:07:27.193 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54015.log 2023-12-05T23:07:27.193 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68091.log.gz 2023-12-05T23:07:27.194 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68649.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68649.log.gz 2023-12-05T23:07:27.195 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56020.log 2023-12-05T23:07:27.195 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40248.log 2023-12-05T23:07:27.196 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56020.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56020.log.gz 2023-12-05T23:07:27.197 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51343.log 2023-12-05T23:07:27.197 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.40248.log: 74.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40248.log.gz 2023-12-05T23:07:27.198 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53941.log 2023-12-05T23:07:27.198 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51343.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51343.log.gz 2023-12-05T23:07:27.199 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48160.log 2023-12-05T23:07:27.201 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53941.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53941.log.gz 2023-12-05T23:07:27.201 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46222.log 2023-12-05T23:07:27.202 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48160.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48160.log.gz 2023-12-05T23:07:27.203 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35003.log 2023-12-05T23:07:27.203 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46222.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46222.log.gz 2023-12-05T23:07:27.204 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50763.log 2023-12-05T23:07:27.204 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.35003.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35003.log.gz 2023-12-05T23:07:27.205 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39600.log 2023-12-05T23:07:27.205 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50763.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50763.log.gz 2023-12-05T23:07:27.206 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64893.log 2023-12-05T23:07:27.206 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39600.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39600.log.gz 2023-12-05T23:07:27.207 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50852.log 2023-12-05T23:07:27.207 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64893.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64893.log.gz 2023-12-05T23:07:27.208 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.65844.log 2023-12-05T23:07:27.208 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50852.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50852.log.gz 2023-12-05T23:07:27.209 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49810.log 2023-12-05T23:07:27.220 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.2.65844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53576.log 2023-12-05T23:07:27.221 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49810.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49810.log.gz 2023-12-05T23:07:27.228 INFO:teuthology.orchestra.run.smithi190.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.65844.log.gz 2023-12-05T23:07:27.228 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66871.log 2023-12-05T23:07:27.229 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53576.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53576.log.gz 2023-12-05T23:07:27.230 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57864.log 2023-12-05T23:07:27.231 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66871.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66871.log.gz 2023-12-05T23:07:27.231 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67284.log 2023-12-05T23:07:27.232 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57864.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57864.log.gz 2023-12-05T23:07:27.233 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65652.log 2023-12-05T23:07:27.235 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67284.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67284.log.gz 2023-12-05T23:07:27.236 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71392.log 2023-12-05T23:07:27.236 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65652.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65652.log.gz 2023-12-05T23:07:27.237 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43201.log 2023-12-05T23:07:27.237 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71392.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71392.log.gz 2023-12-05T23:07:27.238 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58607.log 2023-12-05T23:07:27.238 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43201.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43201.log.gz 2023-12-05T23:07:27.239 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63784.log 2023-12-05T23:07:27.240 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58607.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58607.log.gz 2023-12-05T23:07:27.240 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54353.log 2023-12-05T23:07:27.240 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63784.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63784.log.gz 2023-12-05T23:07:27.241 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75236.log 2023-12-05T23:07:27.244 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54353.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54353.log.gz 2023-12-05T23:07:27.244 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47833.log 2023-12-05T23:07:27.244 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75236.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75236.log.gz 2023-12-05T23:07:27.244 INFO:teuthology.orchestra.run.smithi190.stderr: 94.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51554.log 2023-12-05T23:07:27.244 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.40239.log.gz 2023-12-05T23:07:27.246 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47833.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54015.log.gz 2023-12-05T23:07:27.247 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56887.log 2023-12-05T23:07:27.247 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69133.log 2023-12-05T23:07:27.248 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56887.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51554.log.gz 2023-12-05T23:07:27.248 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56887.log.gz 2023-12-05T23:07:27.248 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82413.log 2023-12-05T23:07:27.248 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80411.log 2023-12-05T23:07:27.251 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82413.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82413.log.gz 2023-12-05T23:07:27.251 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53480.log 2023-12-05T23:07:27.251 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69133.log.gz 2023-12-05T23:07:27.251 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47833.log.gz 2023-12-05T23:07:27.251 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80411.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80411.log.gz 2023-12-05T23:07:27.251 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81529.log 2023-12-05T23:07:27.252 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73753.log 2023-12-05T23:07:27.252 INFO:teuthology.orchestra.run.smithi190.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53480.log.gz/var/log/ceph/cluster2-client.admin.81529.log: 2023-12-05T23:07:27.252 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68732.log 2023-12-05T23:07:27.252 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81529.log.gz 2023-12-05T23:07:27.252 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62545.log 2023-12-05T23:07:27.252 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73753.log: /var/log/ceph/cluster2-client.mirror.68732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71842.log 2023-12-05T23:07:27.253 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62545.log: 87.5% 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68732.log.gz 2023-12-05T23:07:27.253 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62545.log.gz 2023-12-05T23:07:27.253 INFO:teuthology.orchestra.run.smithi190.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73753.log.gz 2023-12-05T23:07:27.254 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64177.log 2023-12-05T23:07:27.254 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52647.log 2023-12-05T23:07:27.255 INFO:teuthology.orchestra.run.smithi190.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71842.log.gz/var/log/ceph/cluster1-client.mirror.64177.log: 2023-12-05T23:07:27.255 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43968.log 2023-12-05T23:07:27.255 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64177.log.gz 2023-12-05T23:07:27.268 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67637.log 2023-12-05T23:07:27.268 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.43968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55288.log 2023-12-05T23:07:27.270 INFO:teuthology.orchestra.run.smithi190.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.67637.log: -- replaced with /var/log/ceph/cluster1-client.admin.43968.log.gz 2023-12-05T23:07:27.270 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82465.log 2023-12-05T23:07:27.270 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67637.log.gz 2023-12-05T23:07:27.270 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55288.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-12-05T23:07:27.271 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82465.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82465.log.gz 2023-12-05T23:07:27.271 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49442.log 2023-12-05T23:07:27.272 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-mon.a.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55288.log.gz 2023-12-05T23:07:27.272 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62415.log 2023-12-05T23:07:27.273 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49442.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49442.log.gz 2023-12-05T23:07:27.274 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49176.log 2023-12-05T23:07:27.274 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62415.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62415.log.gz 2023-12-05T23:07:27.275 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82960.log 2023-12-05T23:07:27.275 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49176.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49176.log.gz 2023-12-05T23:07:27.276 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52447.log 2023-12-05T23:07:27.276 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82960.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82960.log.gz 2023-12-05T23:07:27.277 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83194.log 2023-12-05T23:07:27.278 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52447.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52447.log.gz 2023-12-05T23:07:27.279 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39834.log 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83194.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83194.log.gz 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77992.log 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52167.log.gz 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39834.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39834.log.gz 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64486.log 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48967.log 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77992.log.gz 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64486.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83090.log 2023-12-05T23:07:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48967.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64486.log.gz 2023-12-05T23:07:27.285 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47661.log 2023-12-05T23:07:27.285 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83090.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83090.log.gz 2023-12-05T23:07:27.285 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55330.log 2023-12-05T23:07:27.290 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47661.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47661.log.gz 2023-12-05T23:07:27.291 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56951.log 2023-12-05T23:07:27.291 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55330.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55330.log.gz 2023-12-05T23:07:27.291 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46797.log 2023-12-05T23:07:27.292 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56951.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56951.log.gz 2023-12-05T23:07:27.293 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49111.log 2023-12-05T23:07:27.298 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46797.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46797.log.gz 2023-12-05T23:07:27.299 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51215.log 2023-12-05T23:07:27.299 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49111.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49111.log.gz 2023-12-05T23:07:27.300 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79527.log 2023-12-05T23:07:27.300 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51215.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51215.log.gz 2023-12-05T23:07:27.301 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78018.log 2023-12-05T23:07:27.301 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79527.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79527.log.gz 2023-12-05T23:07:27.303 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82778.log 2023-12-05T23:07:27.303 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78018.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78018.log.gz 2023-12-05T23:07:27.303 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45212.log 2023-12-05T23:07:27.303 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82778.log.gz 2023-12-05T23:07:27.304 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77368.log 2023-12-05T23:07:27.304 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45212.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45212.log.gz 2023-12-05T23:07:27.305 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83246.log 2023-12-05T23:07:27.305 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77368.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77368.log.gz 2023-12-05T23:07:27.306 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.40335.log 2023-12-05T23:07:27.306 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83246.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83246.log.gz 2023-12-05T23:07:27.307 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40764.log 2023-12-05T23:07:27.320 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.0.40335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53406.log 2023-12-05T23:07:27.321 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.40764.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40764.log.gz 2023-12-05T23:07:27.323 INFO:teuthology.orchestra.run.smithi190.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.40335.log.gz 2023-12-05T23:07:27.340 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43663.log 2023-12-05T23:07:27.341 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53406.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53406.log.gz 2023-12-05T23:07:27.341 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54634.log 2023-12-05T23:07:27.342 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43663.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43663.log.gz 2023-12-05T23:07:27.343 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59751.log 2023-12-05T23:07:27.345 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54634.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54634.log.gz 2023-12-05T23:07:27.345 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67471.log 2023-12-05T23:07:27.346 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59751.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59751.log.gz 2023-12-05T23:07:27.347 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71437.log 2023-12-05T23:07:27.349 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67471.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67471.log.gz 2023-12-05T23:07:27.350 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77810.log 2023-12-05T23:07:27.350 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71437.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71437.log.gz 2023-12-05T23:07:27.351 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42783.log 2023-12-05T23:07:27.351 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77810.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77810.log.gz 2023-12-05T23:07:27.352 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77186.log 2023-12-05T23:07:27.358 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42783.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42783.log.gz 2023-12-05T23:07:27.358 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82153.log 2023-12-05T23:07:27.359 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77186.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77186.log.gz 2023-12-05T23:07:27.359 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57643.log 2023-12-05T23:07:27.360 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82153.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82153.log.gz 2023-12-05T23:07:27.360 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79059.log 2023-12-05T23:07:27.361 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57643.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57643.log.gz 2023-12-05T23:07:27.362 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81919.log 2023-12-05T23:07:27.362 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79059.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79059.log.gz 2023-12-05T23:07:27.363 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78695.log 2023-12-05T23:07:27.363 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81919.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81919.log.gz 2023-12-05T23:07:27.364 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.49336.log 2023-12-05T23:07:27.364 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78695.log.gz 2023-12-05T23:07:27.365 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67057.log 2023-12-05T23:07:27.380 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.2.49336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70785.log 2023-12-05T23:07:27.381 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67057.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67057.log.gz 2023-12-05T23:07:27.382 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45340.log.gz 2023-12-05T23:07:27.382 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72067.log 2023-12-05T23:07:27.383 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70785.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70785.log.gz 2023-12-05T23:07:27.384 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.73955.log 2023-12-05T23:07:27.384 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.72067.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72067.log.gz 2023-12-05T23:07:27.385 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73933.log 2023-12-05T23:07:27.388 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.73955.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.73955.log.gz 2023-12-05T23:07:27.388 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46971.log 2023-12-05T23:07:27.389 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73933.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.49336.log.gz 2023-12-05T23:07:27.390 INFO:teuthology.orchestra.run.smithi190.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73933.log.gz 2023-12-05T23:07:27.390 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59187.log 2023-12-05T23:07:27.391 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50076.log 2023-12-05T23:07:27.391 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46971.log: /var/log/ceph/cluster1-client.mirror.59187.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46971.log.gz 2023-12-05T23:07:27.391 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59187.log.gz 2023-12-05T23:07:27.391 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76945.log 2023-12-05T23:07:27.392 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51726.log 2023-12-05T23:07:27.392 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50076.log: /var/log/ceph/cluster2-client.admin.76945.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76945.log.gz 2023-12-05T23:07:27.393 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50076.log.gz 2023-12-05T23:07:27.393 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49054.log 2023-12-05T23:07:27.393 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.48222.log 2023-12-05T23:07:27.394 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51726.log: /var/log/ceph/cluster2-client.mirror.49054.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51726.log.gz 2023-12-05T23:07:27.394 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50911.log 2023-12-05T23:07:27.408 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.3.48222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48040.log 2023-12-05T23:07:27.409 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50911.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43791.log.gz 2023-12-05T23:07:27.410 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51785.log 2023-12-05T23:07:27.411 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48040.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48040.log.gz 2023-12-05T23:07:27.412 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65675.log 2023-12-05T23:07:27.413 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51785.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51785.log.gz 2023-12-05T23:07:27.413 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59164.log 2023-12-05T23:07:27.414 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65675.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65675.log.gz 2023-12-05T23:07:27.415 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42564.log 2023-12-05T23:07:27.415 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59164.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59164.log.gz 2023-12-05T23:07:27.416 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79137.log 2023-12-05T23:07:27.417 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42564.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42564.log.gz 2023-12-05T23:07:27.418 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81399.log 2023-12-05T23:07:27.418 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79137.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79137.log.gz 2023-12-05T23:07:27.419 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60913.log 2023-12-05T23:07:27.419 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81399.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81399.log.gz 2023-12-05T23:07:27.420 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53316.log 2023-12-05T23:07:27.421 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60913.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60913.log.gz 2023-12-05T23:07:27.421 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48180.log 2023-12-05T23:07:27.436 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55956.log 2023-12-05T23:07:27.437 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48180.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48180.log.gz 2023-12-05T23:07:27.438 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47555.log 2023-12-05T23:07:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55956.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55956.log.gz 2023-12-05T23:07:27.456 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51827.log 2023-12-05T23:07:27.462 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47555.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47555.log.gz 2023-12-05T23:07:27.464 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44802.log 2023-12-05T23:07:27.480 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63715.log 2023-12-05T23:07:27.481 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44802.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44802.log.gz 2023-12-05T23:07:27.496 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44717.log 2023-12-05T23:07:27.497 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63715.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63715.log.gz 2023-12-05T23:07:27.504 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50869.log 2023-12-05T23:07:27.512 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44717.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68505.log 2023-12-05T23:07:27.513 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50869.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50869.log.gz 2023-12-05T23:07:27.516 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44717.log.gz 2023-12-05T23:07:27.528 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53004.log 2023-12-05T23:07:27.529 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68505.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68505.log.gz 2023-12-05T23:07:27.536 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62129.log 2023-12-05T23:07:27.536 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53004.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53004.log.gz 2023-12-05T23:07:27.544 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61624.log 2023-12-05T23:07:27.544 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62129.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62129.log.gz 2023-12-05T23:07:27.552 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53369.log 2023-12-05T23:07:27.553 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61624.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61624.log.gz 2023-12-05T23:07:27.560 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40106.log 2023-12-05T23:07:27.560 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53369.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53369.log.gz 2023-12-05T23:07:27.564 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47684.log 2023-12-05T23:07:27.565 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.40106.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40106.log.gz 2023-12-05T23:07:27.565 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73774.log 2023-12-05T23:07:27.571 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47684.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47684.log.gz 2023-12-05T23:07:27.572 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51316.log 2023-12-05T23:07:27.584 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73774.log: 80.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42174.log 2023-12-05T23:07:27.588 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51316.log: -- replaced with /var/log/ceph/cluster2-client.mirror.73774.log.gz 2023-12-05T23:07:27.596 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47446.log 2023-12-05T23:07:27.598 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42174.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42174.log.gz 2023-12-05T23:07:27.612 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46176.log 2023-12-05T23:07:27.624 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47446.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64507.log 2023-12-05T23:07:27.625 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46176.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46176.log.gz 2023-12-05T23:07:27.632 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47446.log.gz 2023-12-05T23:07:27.637 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50911.log.gz 2023-12-05T23:07:27.648 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44737.log 2023-12-05T23:07:27.650 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64507.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64507.log.gz 2023-12-05T23:07:27.668 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40156.log 2023-12-05T23:07:27.680 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44737.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.40353.log 2023-12-05T23:07:27.680 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.40156.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40156.log.gz 2023-12-05T23:07:27.686 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48831.log.gz 2023-12-05T23:07:27.688 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44737.log.gz 2023-12-05T23:07:27.688 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58744.log 2023-12-05T23:07:27.690 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.2.40353.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.40353.log.gz 2023-12-05T23:07:27.691 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34955.log 2023-12-05T23:07:27.692 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58744.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58744.log.gz 2023-12-05T23:07:27.693 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75921.log 2023-12-05T23:07:27.693 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.34955.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34955.log.gz 2023-12-05T23:07:27.694 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59030.log 2023-12-05T23:07:27.694 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75921.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75921.log.gz 2023-12-05T23:07:27.695 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59308.log 2023-12-05T23:07:27.704 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41199.log 2023-12-05T23:07:27.706 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59308.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59308.log.gz 2023-12-05T23:07:27.720 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57275.log 2023-12-05T23:07:27.721 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41199.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41199.log.gz 2023-12-05T23:07:27.721 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43480.log 2023-12-05T23:07:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57275.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57275.log.gz 2023-12-05T23:07:27.723 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51827.log.gz 2023-12-05T23:07:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53828.log 2023-12-05T23:07:27.735 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43480.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59030.log.gz 2023-12-05T23:07:27.735 INFO:teuthology.orchestra.run.smithi190.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43480.log.gz 2023-12-05T23:07:27.736 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48772.log 2023-12-05T23:07:27.737 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53828.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53828.log.gz 2023-12-05T23:07:27.737 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67575.log 2023-12-05T23:07:27.738 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48772.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48772.log.gz 2023-12-05T23:07:27.738 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66726.log 2023-12-05T23:07:27.739 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67575.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67575.log.gz 2023-12-05T23:07:27.739 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62060.log 2023-12-05T23:07:27.742 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66726.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52647.log.gz 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53316.log.gz 2023-12-05T23:07:27.742 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:27.742 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66726.log.gz 2023-12-05T23:07:27.743 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78252.log 2023-12-05T23:07:27.744 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59843.log 2023-12-05T23:07:27.744 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4%/var/log/ceph/cluster2-client.admin.78252.log: -- replaced with /var/log/ceph/cluster2-client.mirror.62060.log.gz 2023-12-05T23:07:27.744 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78252.log.gz 2023-12-05T23:07:27.746 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71190.log 2023-12-05T23:07:27.746 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34979.log 2023-12-05T23:07:27.746 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59843.log.gz/var/log/ceph/cluster1-client.mirror.71190.log: 2023-12-05T23:07:27.746 INFO:teuthology.orchestra.run.smithi190.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71190.log.gz 2023-12-05T23:07:27.746 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63346.log 2023-12-05T23:07:27.746 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.34979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44068.log 2023-12-05T23:07:27.746 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34979.log.gz 2023-12-05T23:07:27.747 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71550.log 2023-12-05T23:07:27.747 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44068.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63346.log.gz 2023-12-05T23:07:27.748 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44604.log 2023-12-05T23:07:27.748 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71550.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44068.log.gz 2023-12-05T23:07:27.748 INFO:teuthology.orchestra.run.smithi190.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71550.log.gz 2023-12-05T23:07:27.749 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60410.log 2023-12-05T23:07:27.749 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43354.log 2023-12-05T23:07:27.749 INFO:teuthology.orchestra.run.smithi190.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44604.log.gz 2023-12-05T23:07:27.749 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60410.log: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80541.log 2023-12-05T23:07:27.749 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60410.log.gz 2023-12-05T23:07:27.752 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67947.log 2023-12-05T23:07:27.753 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80541.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80541.log.gz 2023-12-05T23:07:27.753 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69670.log 2023-12-05T23:07:27.754 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67947.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67947.log.gz 2023-12-05T23:07:27.754 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73226.log 2023-12-05T23:07:27.755 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4%/var/log/ceph/cluster2-client.mirror.69670.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43354.log.gz 2023-12-05T23:07:27.755 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63600.log 2023-12-05T23:07:27.756 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73226.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73226.log.gz 2023-12-05T23:07:27.756 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55031.log 2023-12-05T23:07:27.757 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63600.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69670.log.gz 2023-12-05T23:07:27.757 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52467.log 2023-12-05T23:07:27.759 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55031.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55031.log.gz 2023-12-05T23:07:27.759 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63600.log.gz 2023-12-05T23:07:27.759 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52489.log 2023-12-05T23:07:27.759 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49032.log 2023-12-05T23:07:27.760 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52489.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52467.log.gz 2023-12-05T23:07:27.760 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56221.log 2023-12-05T23:07:27.761 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49032.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49032.log.gz 2023-12-05T23:07:27.761 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.43813.log 2023-12-05T23:07:27.764 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56221.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56221.log.gz 2023-12-05T23:07:27.764 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68587.log 2023-12-05T23:07:27.769 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.3.43813.log: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.43813.log.gz 2023-12-05T23:07:27.769 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78382.log 2023-12-05T23:07:27.772 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68587.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68587.log.gz 2023-12-05T23:07:27.772 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42805.log 2023-12-05T23:07:27.773 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78382.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78382.log.gz 2023-12-05T23:07:27.773 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67823.log 2023-12-05T23:07:27.774 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42805.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42805.log.gz 2023-12-05T23:07:27.775 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79657.log 2023-12-05T23:07:27.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67823.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67823.log.gz 2023-12-05T23:07:27.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44916.log 2023-12-05T23:07:27.777 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79657.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79657.log.gz 2023-12-05T23:07:27.777 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57054.log 2023-12-05T23:07:27.778 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44916.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44916.log.gz 2023-12-05T23:07:27.778 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63622.log 2023-12-05T23:07:27.779 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57054.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57054.log.gz 2023-12-05T23:07:27.779 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67533.log 2023-12-05T23:07:27.781 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63622.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63622.log.gz 2023-12-05T23:07:27.781 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82439.log 2023-12-05T23:07:27.783 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67533.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.3.48222.log.gz 2023-12-05T23:07:27.783 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60173.log 2023-12-05T23:07:27.784 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82439.log: 0.0% 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67533.log.gz 2023-12-05T23:07:27.784 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.82439.log.gz 2023-12-05T23:07:27.784 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55457.log 2023-12-05T23:07:27.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69567.log 2023-12-05T23:07:27.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55457.log: 87.4% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55457.log.gz 2023-12-05T23:07:27.786 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60173.log.gz 2023-12-05T23:07:27.786 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53640.log 2023-12-05T23:07:27.786 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78356.log 2023-12-05T23:07:27.787 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69567.log: /var/log/ceph/cluster2-client.mirror.53640.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53640.log.gz 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69567.log.gz 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79267.log 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51748.log 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78356.log: /var/log/ceph/cluster2-client.admin.79267.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78356.log.gz 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79267.log.gz 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75579.log 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70987.log 2023-12-05T23:07:27.791 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51748.log: /var/log/ceph/cluster1-client.mirror.75579.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75579.log.gz 2023-12-05T23:07:27.795 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49054.log.gz 2023-12-05T23:07:27.800 INFO:teuthology.orchestra.run.smithi062.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-12-05T23:07:27.804 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68773.log 2023-12-05T23:07:27.804 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70560.log 2023-12-05T23:07:27.805 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70987.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70987.log.gz/var/log/ceph/cluster2-client.mirror.68773.log: 2023-12-05T23:07:27.805 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42068.log 2023-12-05T23:07:27.806 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70560.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70560.log.gz 2023-12-05T23:07:27.806 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63211.log 2023-12-05T23:07:27.807 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.42068.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68773.log.gz 2023-12-05T23:07:27.807 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.42068.log.gz 2023-12-05T23:07:27.808 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42150.log 2023-12-05T23:07:27.808 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52509.log 2023-12-05T23:07:27.809 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63211.log: /var/log/ceph/cluster1-client.mirror.42150.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42150.log.gz 2023-12-05T23:07:27.809 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63211.log.gz 2023-12-05T23:07:27.809 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79631.log 2023-12-05T23:07:27.810 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50407.log 2023-12-05T23:07:27.810 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52509.log: /var/log/ceph/cluster2-client.admin.79631.log: 0.0% 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52509.log.gz 2023-12-05T23:07:27.810 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.79631.log.gz 2023-12-05T23:07:27.810 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39488.log 2023-12-05T23:07:27.811 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70830.log 2023-12-05T23:07:27.811 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50407.log: /var/log/ceph/cluster2-client.admin.39488.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39488.log.gz 2023-12-05T23:07:27.820 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65844.log 2023-12-05T23:07:27.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70830.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70830.log.gz 2023-12-05T23:07:27.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55693.log 2023-12-05T23:07:27.822 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65844.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65844.log.gz 2023-12-05T23:07:27.823 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71302.log 2023-12-05T23:07:27.824 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55693.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55693.log.gz 2023-12-05T23:07:27.824 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69216.log 2023-12-05T23:07:27.825 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71302.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71302.log.gz 2023-12-05T23:07:27.825 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59708.log 2023-12-05T23:07:27.826 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69216.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69216.log.gz 2023-12-05T23:07:27.826 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70717.log 2023-12-05T23:07:27.828 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59708.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59708.log.gz 2023-12-05T23:07:27.828 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66088.log 2023-12-05T23:07:27.829 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70717.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70717.log.gz 2023-12-05T23:07:27.829 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49832.log 2023-12-05T23:07:27.831 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66088.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66088.log.gz 2023-12-05T23:07:27.832 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57100.log 2023-12-05T23:07:27.838 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49832.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48967.log.gz 2023-12-05T23:07:27.838 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82257.log 2023-12-05T23:07:27.839 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57100.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57100.log.gz 2023-12-05T23:07:27.839 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50098.log 2023-12-05T23:07:27.840 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82257.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82257.log.gz 2023-12-05T23:07:27.841 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45479.log 2023-12-05T23:07:27.856 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40544.log 2023-12-05T23:07:27.857 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45479.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45479.log.gz 2023-12-05T23:07:27.873 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78981.log 2023-12-05T23:07:27.873 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40544.log.gz 2023-12-05T23:07:27.888 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42518.log 2023-12-05T23:07:27.889 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78981.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78981.log.gz 2023-12-05T23:07:27.896 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79865.log 2023-12-05T23:07:27.897 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42518.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42518.log.gz 2023-12-05T23:07:27.904 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45166.log 2023-12-05T23:07:27.904 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79865.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79865.log.gz 2023-12-05T23:07:27.913 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63447.log 2023-12-05T23:07:27.913 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45166.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45166.log.gz 2023-12-05T23:07:27.920 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49605.log 2023-12-05T23:07:27.921 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63447.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63447.log.gz 2023-12-05T23:07:27.928 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45318.log 2023-12-05T23:07:27.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.49605.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49605.log.gz 2023-12-05T23:07:27.936 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65149.log 2023-12-05T23:07:27.937 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45318.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45318.log.gz 2023-12-05T23:07:27.944 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63119.log 2023-12-05T23:07:27.944 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65149.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65149.log.gz 2023-12-05T23:07:27.952 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58786.log 2023-12-05T23:07:27.954 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63119.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63119.log.gz 2023-12-05T23:07:27.960 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53517.log 2023-12-05T23:07:27.961 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58786.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58786.log.gz 2023-12-05T23:07:27.965 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52489.log.gz 2023-12-05T23:07:27.965 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50514.log 2023-12-05T23:07:27.966 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53517.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53517.log.gz 2023-12-05T23:07:27.967 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59820.log 2023-12-05T23:07:27.967 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50514.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50514.log.gz 2023-12-05T23:07:27.967 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62436.log 2023-12-05T23:07:27.968 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59820.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59820.log.gz 2023-12-05T23:07:27.968 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80697.log 2023-12-05T23:07:27.971 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62436.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62436.log.gz 2023-12-05T23:07:27.971 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76477.log 2023-12-05T23:07:27.972 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80697.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80697.log.gz 2023-12-05T23:07:27.972 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53747.log 2023-12-05T23:07:27.973 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76477.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76477.log.gz 2023-12-05T23:07:27.973 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51257.log 2023-12-05T23:07:27.974 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53747.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53747.log.gz 2023-12-05T23:07:27.975 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.40338.log 2023-12-05T23:07:27.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51257.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51257.log.gz 2023-12-05T23:07:27.976 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76612.log 2023-12-05T23:07:27.981 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.40338.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.40338.log.gz 2023-12-05T23:07:27.981 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48945.log 2023-12-05T23:07:27.982 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76612.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76612.log.gz 2023-12-05T23:07:27.982 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39956.log 2023-12-05T23:07:27.983 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48945.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48945.log.gz 2023-12-05T23:07:27.983 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69774.log 2023-12-05T23:07:27.984 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39956.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39956.log.gz 2023-12-05T23:07:27.984 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59598.log 2023-12-05T23:07:27.985 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69774.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69774.log.gz 2023-12-05T23:07:27.985 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68422.log 2023-12-05T23:07:27.986 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59598.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59598.log.gz 2023-12-05T23:07:27.987 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82804.log 2023-12-05T23:07:27.988 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68422.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68422.log.gz 2023-12-05T23:07:27.989 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45189.log 2023-12-05T23:07:27.989 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82804.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82804.log.gz 2023-12-05T23:07:27.989 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47899.log 2023-12-05T23:07:27.990 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45189.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45189.log.gz 2023-12-05T23:07:27.993 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54394.log 2023-12-05T23:07:27.994 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47899.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47899.log.gz 2023-12-05T23:07:27.994 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81607.log 2023-12-05T23:07:27.994 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54394.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54394.log.gz 2023-12-05T23:07:27.994 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64572.log 2023-12-05T23:07:27.994 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81607.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81607.log.gz 2023-12-05T23:07:27.995 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52108.log 2023-12-05T23:07:27.997 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64572.log: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64572.log.gz 2023-12-05T23:07:27.997 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82101.log 2023-12-05T23:07:27.997 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52108.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52108.log.gz 2023-12-05T23:07:27.998 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62216.log 2023-12-05T23:07:27.998 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82101.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82101.log.gz 2023-12-05T23:07:27.998 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69030.log 2023-12-05T23:07:28.000 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62216.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62216.log.gz 2023-12-05T23:07:28.001 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63323.log 2023-12-05T23:07:28.002 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69030.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69030.log.gz 2023-12-05T23:07:28.002 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.51598.log 2023-12-05T23:07:28.003 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63323.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63323.log.gz 2023-12-05T23:07:28.003 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43426.log 2023-12-05T23:07:28.012 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.2.51598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56454.log 2023-12-05T23:07:28.012 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.43426.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43426.log.gz 2023-12-05T23:07:28.018 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48309.log 2023-12-05T23:07:28.020 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56454.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56454.log.gz 2023-12-05T23:07:28.020 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.51598.log.gz 2023-12-05T23:07:28.020 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76221.log 2023-12-05T23:07:28.022 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48309.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48309.log.gz 2023-12-05T23:07:28.022 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73513.log 2023-12-05T23:07:28.023 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76221.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76221.log.gz 2023-12-05T23:07:28.023 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52125.log 2023-12-05T23:07:28.024 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73513.log: 86.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73513.log.gz 2023-12-05T23:07:28.024 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77160.log 2023-12-05T23:07:28.025 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52125.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52125.log.gz 2023-12-05T23:07:28.026 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50780.log 2023-12-05T23:07:28.026 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77160.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77160.log.gz 2023-12-05T23:07:28.027 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.56557.log 2023-12-05T23:07:28.028 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50780.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50780.log.gz 2023-12-05T23:07:28.028 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75641.log 2023-12-05T23:07:28.048 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.1.56557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64096.log 2023-12-05T23:07:28.050 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75641.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75641.log.gz 2023-12-05T23:07:28.053 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50098.log.gz 2023-12-05T23:07:28.054 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62196.log 2023-12-05T23:07:28.055 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64096.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64096.log.gz 2023-12-05T23:07:28.056 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73268.log 2023-12-05T23:07:28.058 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62196.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62196.log.gz 2023-12-05T23:07:28.058 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63139.log 2023-12-05T23:07:28.059 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73268.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73268.log.gz 2023-12-05T23:07:28.059 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48584.log 2023-12-05T23:07:28.061 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63139.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63139.log.gz 2023-12-05T23:07:28.061 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70875.log 2023-12-05T23:07:28.064 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48584.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48584.log.gz 2023-12-05T23:07:28.064 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44203.log 2023-12-05T23:07:28.064 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70875.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70875.log.gz 2023-12-05T23:07:28.064 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65629.log 2023-12-05T23:07:28.065 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44203.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44203.log.gz 2023-12-05T23:07:28.065 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62637.log 2023-12-05T23:07:28.066 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65629.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65629.log.gz 2023-12-05T23:07:28.066 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57252.log 2023-12-05T23:07:28.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62637.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62637.log.gz 2023-12-05T23:07:28.067 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48482.log 2023-12-05T23:07:28.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57252.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57252.log.gz 2023-12-05T23:07:28.068 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60502.log 2023-12-05T23:07:28.069 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48482.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48482.log.gz 2023-12-05T23:07:28.069 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60546.log 2023-12-05T23:07:28.070 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60502.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60502.log.gz 2023-12-05T23:07:28.070 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53083.log 2023-12-05T23:07:28.072 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60546.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60546.log.gz 2023-12-05T23:07:28.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52825.log 2023-12-05T23:07:28.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53083.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53083.log.gz 2023-12-05T23:07:28.074 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-12-05T23:07:28.074 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52825.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52825.log.gz 2023-12-05T23:07:28.075 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53021.log 2023-12-05T23:07:28.079 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2.audit.log: 92.5% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-12-05T23:07:28.080 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53661.log 2023-12-05T23:07:28.082 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53021.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53021.log.gz 2023-12-05T23:07:28.082 INFO:teuthology.orchestra.run.smithi190.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.56557.log.gz 2023-12-05T23:07:28.082 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74113.log 2023-12-05T23:07:28.083 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75349.log 2023-12-05T23:07:28.083 INFO:teuthology.orchestra.run.smithi190.stderr: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53661.log.gz 2023-12-05T23:07:28.084 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76635.log 2023-12-05T23:07:28.084 INFO:teuthology.orchestra.run.smithi190.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74113.log.gz 2023-12-05T23:07:28.084 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75349.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75349.log.gz 2023-12-05T23:07:28.084 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45503.log 2023-12-05T23:07:28.085 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63805.log 2023-12-05T23:07:28.085 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45503.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76635.log.gz 2023-12-05T23:07:28.085 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42849.log 2023-12-05T23:07:28.086 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63805.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45503.log.gz 2023-12-05T23:07:28.086 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63805.log.gz 2023-12-05T23:07:28.088 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43617.log 2023-12-05T23:07:28.088 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51294.log 2023-12-05T23:07:28.088 INFO:teuthology.orchestra.run.smithi190.stderr: 81.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42849.log.gz 2023-12-05T23:07:28.088 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60216.log 2023-12-05T23:07:28.088 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43617.log.gz 2023-12-05T23:07:28.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51294.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51294.log.gz 2023-12-05T23:07:28.089 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76750.log 2023-12-05T23:07:28.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70312.log 2023-12-05T23:07:28.091 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76750.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60216.log.gz 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76750.log.gz 2023-12-05T23:07:28.091 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:28.091 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80255.log 2023-12-05T23:07:28.092 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58723.log 2023-12-05T23:07:28.092 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70312.log: /var/log/ceph/cluster2-client.admin.80255.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80255.log.gz 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70312.log.gz 2023-12-05T23:07:28.092 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:28.092 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48434.log 2023-12-05T23:07:28.094 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52404.log 2023-12-05T23:07:28.095 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58723.log: /var/log/ceph/cluster1-client.mirror.48434.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48434.log.gz 2023-12-05T23:07:28.095 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58723.log.gz 2023-12-05T23:07:28.095 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57031.log 2023-12-05T23:07:28.095 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71887.log 2023-12-05T23:07:28.095 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52404.log: /var/log/ceph/cluster2-client.mirror.57031.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57031.log.gz 2023-12-05T23:07:28.104 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66260.log 2023-12-05T23:07:28.105 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71887.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71887.log.gz 2023-12-05T23:07:28.105 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70425.log 2023-12-05T23:07:28.107 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66260.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66260.log.gz 2023-12-05T23:07:28.108 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74581.log 2023-12-05T23:07:28.108 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70425.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70425.log.gz 2023-12-05T23:07:28.109 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47231.log 2023-12-05T23:07:28.110 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74581.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74581.log.gz 2023-12-05T23:07:28.110 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.64572.log 2023-12-05T23:07:28.115 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47231.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51316.log.gz 2023-12-05T23:07:28.116 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47231.log.gz 2023-12-05T23:07:28.116 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79839.log 2023-12-05T23:07:28.117 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-12-05T23:07:28.117 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.0.64572.log: /var/log/ceph/cluster2-client.admin.79839.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79839.log.gz 2023-12-05T23:07:28.124 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59007.log 2023-12-05T23:07:28.130 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2.log: 87.8% -- replaced with /var/log/ceph/cluster2.log.gz 2023-12-05T23:07:28.131 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67409.log 2023-12-05T23:07:28.132 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59007.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59007.log.gz 2023-12-05T23:07:28.132 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56241.log 2023-12-05T23:07:28.135 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67409.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67409.log.gz 2023-12-05T23:07:28.136 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62300.log 2023-12-05T23:07:28.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56241.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56241.log.gz 2023-12-05T23:07:28.138 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76177.log 2023-12-05T23:07:28.139 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62300.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62300.log.gz 2023-12-05T23:07:28.139 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82622.log 2023-12-05T23:07:28.140 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76177.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76177.log.gz 2023-12-05T23:07:28.140 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54374.log 2023-12-05T23:07:28.141 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82622.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82622.log.gz 2023-12-05T23:07:28.141 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79943.log 2023-12-05T23:07:28.143 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54374.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54374.log.gz 2023-12-05T23:07:28.144 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40585.log 2023-12-05T23:07:28.145 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79943.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79943.log.gz 2023-12-05T23:07:28.145 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47959.log 2023-12-05T23:07:28.145 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40585.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.40585.log.gz 2023-12-05T23:07:28.146 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44287.log 2023-12-05T23:07:28.147 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47959.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47959.log.gz 2023-12-05T23:07:28.147 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61305.log 2023-12-05T23:07:28.153 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44287.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44287.log.gz 2023-12-05T23:07:28.154 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43528.log 2023-12-05T23:07:28.155 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61305.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61305.log.gz 2023-12-05T23:07:28.155 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49213.log 2023-12-05T23:07:28.157 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43528.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43528.log.gz 2023-12-05T23:07:28.157 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47770.log 2023-12-05T23:07:28.158 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49213.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49213.log.gz 2023-12-05T23:07:28.158 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71235.log 2023-12-05T23:07:28.159 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47770.log: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.64572.log.gz 2023-12-05T23:07:28.164 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47770.log.gz 2023-12-05T23:07:28.164 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58630.log 2023-12-05T23:07:28.165 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71212.log 2023-12-05T23:07:28.166 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71235.log: /var/log/ceph/cluster1-client.mirror.58630.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71235.log.gz 2023-12-05T23:07:28.166 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58630.log.gz 2023-12-05T23:07:28.166 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70627.log 2023-12-05T23:07:28.166 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71212.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71212.log.gz 2023-12-05T23:07:28.166 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81165.log 2023-12-05T23:07:28.167 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47727.log 2023-12-05T23:07:28.167 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70627.log: /var/log/ceph/cluster2-client.admin.81165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81165.log.gz 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70627.log.gz 2023-12-05T23:07:28.168 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:28.168 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65719.log 2023-12-05T23:07:28.168 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61084.log 2023-12-05T23:07:28.170 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47727.log: /var/log/ceph/cluster2-client.mirror.65719.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65719.log.gz 2023-12-05T23:07:28.173 INFO:teuthology.orchestra.run.smithi190.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47727.log.gz 2023-12-05T23:07:28.174 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61744.log 2023-12-05T23:07:28.174 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42240.log 2023-12-05T23:07:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61084.log: /var/log/ceph/cluster1-client.mirror.61744.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61084.log.gz 2023-12-05T23:07:28.175 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61744.log.gz 2023-12-05T23:07:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62525.log 2023-12-05T23:07:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66788.log 2023-12-05T23:07:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42240.log: /var/log/ceph/cluster2-client.mirror.62525.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42240.log.gz 2023-12-05T23:07:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47317.log 2023-12-05T23:07:28.177 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62525.log.gz 2023-12-05T23:07:28.179 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66788.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66788.log.gz 2023-12-05T23:07:28.179 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44331.log 2023-12-05T23:07:28.180 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81685.log 2023-12-05T23:07:28.185 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47317.log: /var/log/ceph/cluster2-client.mirror.44331.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47317.log.gz 2023-12-05T23:07:28.185 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55544.log 2023-12-05T23:07:28.186 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81685.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81685.log.gz 2023-12-05T23:07:28.186 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63387.log 2023-12-05T23:07:28.189 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55544.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55544.log.gz 2023-12-05T23:07:28.189 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54919.log 2023-12-05T23:07:28.191 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63387.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63387.log.gz 2023-12-05T23:07:28.192 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69112.log 2023-12-05T23:07:28.193 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54919.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54919.log.gz 2023-12-05T23:07:28.193 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64074.log 2023-12-05T23:07:28.197 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69112.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69112.log.gz 2023-12-05T23:07:28.198 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43684.log 2023-12-05T23:07:28.199 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64074.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64074.log.gz 2023-12-05T23:07:28.199 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44111.log 2023-12-05T23:07:28.205 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43684.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43684.log.gz 2023-12-05T23:07:28.205 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42418.log 2023-12-05T23:07:28.206 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44111.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44111.log.gz 2023-12-05T23:07:28.206 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58944.log 2023-12-05T23:07:28.208 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42418.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42418.log.gz 2023-12-05T23:07:28.208 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48200.log 2023-12-05T23:07:28.209 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58944.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58944.log.gz 2023-12-05T23:07:28.209 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65282.log 2023-12-05T23:07:28.211 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48200.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48200.log.gz 2023-12-05T23:07:28.211 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52783.log 2023-12-05T23:07:28.213 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65282.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65282.log.gz 2023-12-05T23:07:28.214 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82491.log 2023-12-05T23:07:28.214 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51748.log.gz 2023-12-05T23:07:28.214 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52783.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52783.log.gz 2023-12-05T23:07:28.215 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48120.log 2023-12-05T23:07:28.215 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51455.log 2023-12-05T23:07:28.215 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82491.log: /var/log/ceph/cluster2-client.mirror.48120.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82491.log.gz 2023-12-05T23:07:28.216 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48392.log 2023-12-05T23:07:28.216 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51455.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48120.log.gz 2023-12-05T23:07:28.217 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51455.log.gz 2023-12-05T23:07:28.217 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67678.log 2023-12-05T23:07:28.218 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81971.log 2023-12-05T23:07:28.218 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48392.log: /var/log/ceph/cluster2-client.mirror.67678.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48392.log.gz 2023-12-05T23:07:28.219 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44022.log 2023-12-05T23:07:28.219 INFO:teuthology.orchestra.run.smithi190.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67678.log.gz 2023-12-05T23:07:28.219 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81971.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81971.log.gz 2023-12-05T23:07:28.219 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75721.log 2023-12-05T23:07:28.220 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60105.log 2023-12-05T23:07:28.221 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75721.log: /var/log/ceph/cluster2-client.mirror.44022.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44022.log.gz 2023-12-05T23:07:28.221 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60890.log 2023-12-05T23:07:28.224 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60105.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60105.log.gz 2023-12-05T23:07:28.224 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64287.log 2023-12-05T23:07:28.225 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60890.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60890.log.gz 2023-12-05T23:07:28.225 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46153.log 2023-12-05T23:07:28.227 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64287.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64287.log.gz 2023-12-05T23:07:28.228 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41165.log 2023-12-05T23:07:28.229 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46153.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46153.log.gz 2023-12-05T23:07:28.229 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51886.log 2023-12-05T23:07:28.230 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41165.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41165.log.gz 2023-12-05T23:07:28.230 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73889.log 2023-12-05T23:07:28.231 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51886.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51886.log.gz 2023-12-05T23:07:28.231 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83064.log 2023-12-05T23:07:28.232 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73889.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73889.log.gz 2023-12-05T23:07:28.232 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69484.log 2023-12-05T23:07:28.233 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83064.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83064.log.gz 2023-12-05T23:07:28.233 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57488.log 2023-12-05T23:07:28.236 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69484.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69484.log.gz 2023-12-05T23:07:28.237 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61532.log 2023-12-05T23:07:28.237 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57488.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57488.log.gz 2023-12-05T23:07:28.238 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40738.log 2023-12-05T23:07:28.240 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61532.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61532.log.gz 2023-12-05T23:07:28.240 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52805.log 2023-12-05T23:07:28.241 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.40738.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40738.log.gz 2023-12-05T23:07:28.241 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50056.log 2023-12-05T23:07:28.248 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47017.log 2023-12-05T23:07:28.251 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50407.log.gz 2023-12-05T23:07:28.252 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50056.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50056.log.gz 2023-12-05T23:07:28.253 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77758.log 2023-12-05T23:07:28.253 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47017.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47017.log.gz 2023-12-05T23:07:28.254 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69319.log 2023-12-05T23:07:28.254 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77758.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77758.log.gz 2023-12-05T23:07:28.255 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68926.log 2023-12-05T23:07:28.256 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69319.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69319.log.gz 2023-12-05T23:07:28.256 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56692.log 2023-12-05T23:07:28.259 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68926.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68926.log.gz 2023-12-05T23:07:28.259 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73291.log 2023-12-05T23:07:28.263 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56692.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56692.log.gz 2023-12-05T23:07:28.264 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50303.log 2023-12-05T23:07:28.264 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73291.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73291.log.gz 2023-12-05T23:07:28.264 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78096.log 2023-12-05T23:07:28.280 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46707.log 2023-12-05T23:07:28.281 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78096.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78096.log.gz 2023-12-05T23:07:28.281 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73433.log 2023-12-05T23:07:28.282 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46707.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46707.log.gz 2023-12-05T23:07:28.288 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58448.log 2023-12-05T23:07:28.320 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78617.log 2023-12-05T23:07:28.321 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58448.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58448.log.gz 2023-12-05T23:07:28.336 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60959.log 2023-12-05T23:07:28.337 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78617.log.gz 2023-12-05T23:07:28.341 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49832.log.gz 2023-12-05T23:07:28.352 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76021.log 2023-12-05T23:07:28.353 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60959.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60959.log.gz 2023-12-05T23:07:28.368 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59995.log 2023-12-05T23:07:28.369 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76021.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76021.log.gz 2023-12-05T23:07:28.384 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71145.log 2023-12-05T23:07:28.385 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59995.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59995.log.gz 2023-12-05T23:07:28.400 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45276.log 2023-12-05T23:07:28.400 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71145.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71145.log.gz 2023-12-05T23:07:28.408 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77056.log 2023-12-05T23:07:28.416 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45276.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68195.log 2023-12-05T23:07:28.416 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77056.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77056.log.gz 2023-12-05T23:07:28.424 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51945.log 2023-12-05T23:07:28.424 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68195.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68195.log.gz 2023-12-05T23:07:28.428 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45276.log.gz 2023-12-05T23:07:28.432 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40335.log 2023-12-05T23:07:28.433 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51945.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51945.log.gz 2023-12-05T23:07:28.444 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39599.log 2023-12-05T23:07:28.445 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40335.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40335.log.gz 2023-12-05T23:07:28.465 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43594.log 2023-12-05T23:07:28.465 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39599.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39599.log.gz 2023-12-05T23:07:28.470 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52404.log.gz 2023-12-05T23:07:28.472 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76993.log 2023-12-05T23:07:28.473 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43594.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43594.log.gz 2023-12-05T23:07:28.473 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58014.log 2023-12-05T23:07:28.474 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.76993.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76993.log.gz 2023-12-05T23:07:28.474 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49074.log 2023-12-05T23:07:28.475 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58014.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58014.log.gz 2023-12-05T23:07:28.475 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73833.log 2023-12-05T23:07:28.476 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49074.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49074.log.gz 2023-12-05T23:07:28.476 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47512.log 2023-12-05T23:07:28.492 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55844.log 2023-12-05T23:07:28.498 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47512.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47512.log.gz 2023-12-05T23:07:28.504 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51534.log 2023-12-05T23:07:28.505 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55844.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55844.log.gz 2023-12-05T23:07:28.512 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75721.log.gz 2023-12-05T23:07:28.516 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77862.log 2023-12-05T23:07:28.517 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51534.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51534.log.gz 2023-12-05T23:07:28.518 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67905.log 2023-12-05T23:07:28.518 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77862.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77862.log.gz 2023-12-05T23:07:28.519 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70042.log 2023-12-05T23:07:28.521 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67905.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67905.log.gz 2023-12-05T23:07:28.522 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66435.log 2023-12-05T23:07:28.522 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70042.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70042.log.gz 2023-12-05T23:07:28.522 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54501.log 2023-12-05T23:07:28.524 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66435.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66435.log.gz 2023-12-05T23:07:28.524 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42094.log 2023-12-05T23:07:28.525 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54501.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54501.log.gz 2023-12-05T23:07:28.525 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43769.log 2023-12-05T23:07:28.526 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.42094.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42094.log.gz 2023-12-05T23:07:28.527 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45120.log 2023-12-05T23:07:28.528 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43769.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43769.log.gz 2023-12-05T23:07:28.528 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76900.log 2023-12-05T23:07:28.529 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45120.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45120.log.gz 2023-12-05T23:07:28.529 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55647.log 2023-12-05T23:07:28.530 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76900.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76900.log.gz 2023-12-05T23:07:28.530 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50948.log 2023-12-05T23:07:28.531 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55647.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55647.log.gz 2023-12-05T23:07:28.531 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39462.log 2023-12-05T23:07:28.534 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50948.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50948.log.gz 2023-12-05T23:07:28.534 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52805.log.gz 2023-12-05T23:07:28.534 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67181.log 2023-12-05T23:07:28.534 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39462.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39462.log.gz 2023-12-05T23:07:28.534 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60719.log 2023-12-05T23:07:28.535 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74171.log 2023-12-05T23:07:28.536 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60719.log: 89.5% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60719.log.gz 2023-12-05T23:07:28.536 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67181.log.gz 2023-12-05T23:07:28.536 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55910.log 2023-12-05T23:07:28.536 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58185.log 2023-12-05T23:07:28.537 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74171.log: /var/log/ceph/cluster1-client.mirror.55910.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55910.log.gz 2023-12-05T23:07:28.537 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74171.log.gz 2023-12-05T23:07:28.537 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58677.log 2023-12-05T23:07:28.538 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43155.log 2023-12-05T23:07:28.538 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58185.log: /var/log/ceph/cluster1-client.mirror.58677.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58185.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58677.log.gz 2023-12-05T23:07:28.538 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:28.539 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62812.log 2023-12-05T23:07:28.539 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81737.log 2023-12-05T23:07:28.539 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43155.log: /var/log/ceph/cluster1-client.mirror.62812.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62812.log.gz 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43155.log.gz 2023-12-05T23:07:28.539 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:28.539 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52224.log 2023-12-05T23:07:28.540 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63367.log 2023-12-05T23:07:28.540 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81737.log: /var/log/ceph/cluster2-client.mirror.52224.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81737.log.gz 2023-12-05T23:07:28.541 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70942.log 2023-12-05T23:07:28.541 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52224.log.gz 2023-12-05T23:07:28.542 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63367.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63367.log.gz 2023-12-05T23:07:28.543 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59774.log 2023-12-05T23:07:28.544 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48060.log 2023-12-05T23:07:28.544 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70942.log: /var/log/ceph/cluster1-client.mirror.59774.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70942.log.gz 2023-12-05T23:07:28.544 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59774.log.gz 2023-12-05T23:07:28.544 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80905.log 2023-12-05T23:07:28.545 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58317.log 2023-12-05T23:07:28.545 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48060.log: /var/log/ceph/cluster2-client.admin.80905.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80905.log.gz 2023-12-05T23:07:28.546 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48060.log.gz 2023-12-05T23:07:28.546 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71595.log 2023-12-05T23:07:28.547 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71752.log 2023-12-05T23:07:28.547 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58317.log: /var/log/ceph/cluster1-client.mirror.71595.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71595.log.gz 2023-12-05T23:07:28.547 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58317.log.gz 2023-12-05T23:07:28.547 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57468.log 2023-12-05T23:07:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.46503.log 2023-12-05T23:07:28.548 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71752.log: /var/log/ceph/cluster2-client.mirror.57468.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71752.log.gz 2023-12-05T23:07:28.549 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51360.log 2023-12-05T23:07:28.549 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57468.log.gz/var/log/ceph/cluster1-client.mirror.0.46503.log: 2023-12-05T23:07:28.560 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50303.log.gz 2023-12-05T23:07:28.560 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46481.log 2023-12-05T23:07:28.561 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47598.log 2023-12-05T23:07:28.562 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51360.log: /var/log/ceph/cluster2-client.mirror.46481.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51360.log.gz 2023-12-05T23:07:28.562 INFO:teuthology.orchestra.run.smithi190.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46481.log.gz 2023-12-05T23:07:28.563 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69422.log 2023-12-05T23:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.46503.log.gz 2023-12-05T23:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77602.log 2023-12-05T23:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70177.log 2023-12-05T23:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69422.log: /var/log/ceph/cluster2-client.admin.77602.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77602.log.gz 2023-12-05T23:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69422.log.gz 2023-12-05T23:07:28.566 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67492.log 2023-12-05T23:07:28.567 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57778.log 2023-12-05T23:07:28.567 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70177.log: /var/log/ceph/cluster2-client.mirror.67492.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70177.log.gz 2023-12-05T23:07:28.567 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74213.log 2023-12-05T23:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67492.log.gz 2023-12-05T23:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57778.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57778.log.gz 2023-12-05T23:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47598.log.gz 2023-12-05T23:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73374.log 2023-12-05T23:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69608.log 2023-12-05T23:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74213.log: /var/log/ceph/cluster2-client.mirror.73374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64033.log 2023-12-05T23:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr: 80.6%/var/log/ceph/cluster2-client.mirror.69608.log: -- replaced with /var/log/ceph/cluster2-client.mirror.73374.log.gz 2023-12-05T23:07:28.571 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69608.log.gz 2023-12-05T23:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51908.log 2023-12-05T23:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61107.log 2023-12-05T23:07:28.589 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64033.log: /var/log/ceph/cluster2-client.mirror.51908.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64033.log.gz 2023-12-05T23:07:28.589 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60390.log 2023-12-05T23:07:28.590 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61107.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61107.log.gz 2023-12-05T23:07:28.591 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81425.log 2023-12-05T23:07:28.592 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60390.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60390.log.gz 2023-12-05T23:07:28.593 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47979.log 2023-12-05T23:07:28.593 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81425.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81425.log.gz 2023-12-05T23:07:28.593 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68851.log 2023-12-05T23:07:28.595 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47979.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47979.log.gz 2023-12-05T23:07:28.595 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52246.log 2023-12-05T23:07:28.596 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68851.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68851.log.gz 2023-12-05T23:07:28.597 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80151.log 2023-12-05T23:07:28.612 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82648.log 2023-12-05T23:07:28.613 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80151.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80151.log.gz 2023-12-05T23:07:28.620 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75411.log 2023-12-05T23:07:28.621 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82648.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82648.log.gz 2023-12-05T23:07:28.636 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75621.log 2023-12-05T23:07:28.637 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75411.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75411.log.gz 2023-12-05T23:07:28.644 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71955.log 2023-12-05T23:07:28.645 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75621.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44331.log.gz 2023-12-05T23:07:28.645 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75621.log.gz 2023-12-05T23:07:28.645 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78330.log 2023-12-05T23:07:28.646 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71955.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71955.log.gz 2023-12-05T23:07:28.646 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.54015.log 2023-12-05T23:07:28.646 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78330.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78330.log.gz 2023-12-05T23:07:28.647 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68319.log 2023-12-05T23:07:28.665 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.2.54015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65195.log 2023-12-05T23:07:28.665 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68319.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68319.log.gz 2023-12-05T23:07:28.680 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47876.log 2023-12-05T23:07:28.680 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65195.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65195.log.gz 2023-12-05T23:07:28.681 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49790.log 2023-12-05T23:07:28.686 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47876.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47876.log.gz 2023-12-05T23:07:28.700 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74048.log 2023-12-05T23:07:28.701 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49790.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49790.log.gz 2023-12-05T23:07:28.701 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64982.log 2023-12-05T23:07:28.702 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74048.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74048.log.gz 2023-12-05T23:07:28.717 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72000.log 2023-12-05T23:07:28.717 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64982.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64982.log.gz 2023-12-05T23:07:28.724 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61239.log 2023-12-05T23:07:28.724 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.72000.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72000.log.gz 2023-12-05T23:07:28.732 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70965.log 2023-12-05T23:07:28.733 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61239.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61239.log.gz 2023-12-05T23:07:28.740 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79189.log 2023-12-05T23:07:28.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70965.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70965.log.gz 2023-12-05T23:07:28.743 INFO:teuthology.orchestra.run.smithi190.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.54015.log.gz 2023-12-05T23:07:28.743 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66066.log 2023-12-05T23:07:28.744 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79189.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79189.log.gz 2023-12-05T23:07:28.744 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55008.log 2023-12-05T23:07:28.746 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66066.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66066.log.gz 2023-12-05T23:07:28.747 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70897.log 2023-12-05T23:07:28.747 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55008.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55008.log.gz 2023-12-05T23:07:28.748 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43747.log 2023-12-05T23:07:28.748 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70897.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70897.log.gz 2023-12-05T23:07:28.749 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70650.log 2023-12-05T23:07:28.755 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43747.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43747.log.gz 2023-12-05T23:07:28.755 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64417.log 2023-12-05T23:07:28.756 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70650.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70650.log.gz 2023-12-05T23:07:28.756 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61153.log 2023-12-05T23:07:28.757 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64417.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64417.log.gz 2023-12-05T23:07:28.757 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57968.log 2023-12-05T23:07:28.758 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61153.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61153.log.gz 2023-12-05T23:07:28.758 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61831.log 2023-12-05T23:07:28.759 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57968.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57968.log.gz 2023-12-05T23:07:28.759 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59952.log 2023-12-05T23:07:28.761 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61831.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61831.log.gz 2023-12-05T23:07:28.762 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.41949.log 2023-12-05T23:07:28.763 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59952.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59952.log.gz 2023-12-05T23:07:28.763 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40619.log 2023-12-05T23:07:28.765 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.2.41949.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73433.log.gz 2023-12-05T23:07:28.769 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52246.log.gz 2023-12-05T23:07:28.769 INFO:teuthology.orchestra.run.smithi190.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.41949.log.gz 2023-12-05T23:07:28.774 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67926.log 2023-12-05T23:07:28.774 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74746.log 2023-12-05T23:07:28.774 INFO:teuthology.orchestra.run.smithi190.stderr: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40619.log.gz 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80983.log 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47489.log 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74746.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67926.log.gz/var/log/ceph/cluster2-client.admin.80983.log: 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80983.log.gz 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40653.log 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.52009.log 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47489.log: 91.4%/var/log/ceph/cluster1-client.mirror.40653.log: -- replaced with /var/log/ceph/cluster2-client.mirror.74746.log.gz 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40653.log.gz 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62681.log 2023-12-05T23:07:28.775 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.2.52009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67347.log 2023-12-05T23:07:28.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62681.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62681.log.gz 2023-12-05T23:07:28.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75328.log 2023-12-05T23:07:28.778 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67347.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47489.log.gz 2023-12-05T23:07:28.778 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67347.log.gz 2023-12-05T23:07:28.779 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45253.log 2023-12-05T23:07:28.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79995.log 2023-12-05T23:07:28.780 INFO:teuthology.orchestra.run.smithi190.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75328.log.gz 2023-12-05T23:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45253.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45253.log.gz 2023-12-05T23:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69794.log 2023-12-05T23:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47383.log 2023-12-05T23:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79995.log: /var/log/ceph/cluster2-client.mirror.69794.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79995.log.gz 2023-12-05T23:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50931.log 2023-12-05T23:07:28.788 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47383.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69794.log.gz 2023-12-05T23:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47383.log.gz 2023-12-05T23:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49193.log 2023-12-05T23:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56973.log 2023-12-05T23:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50931.log: /var/log/ceph/cluster2-client.mirror.49193.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50931.log.gz 2023-12-05T23:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44627.log 2023-12-05T23:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49193.log.gz 2023-12-05T23:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56973.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56973.log.gz 2023-12-05T23:07:28.795 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58208.log 2023-12-05T23:07:28.795 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47919.log 2023-12-05T23:07:28.796 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44627.log: /var/log/ceph/cluster1-client.mirror.58208.log: 87.4% 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44627.log.gz 2023-12-05T23:07:28.796 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58208.log.gz 2023-12-05T23:07:28.797 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39566.log 2023-12-05T23:07:28.798 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54764.log 2023-12-05T23:07:28.798 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47919.log: /var/log/ceph/cluster2-client.admin.39566.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39566.log.gz 2023-12-05T23:07:28.798 INFO:teuthology.orchestra.run.smithi190.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.52009.log.gz 2023-12-05T23:07:28.798 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47919.log.gz 2023-12-05T23:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.49605.log 2023-12-05T23:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75470.log 2023-12-05T23:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.49605.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68947.log 2023-12-05T23:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54764.log.gz 2023-12-05T23:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43704.log 2023-12-05T23:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68947.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68947.log.gz 2023-12-05T23:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43178.log 2023-12-05T23:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43704.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43704.log.gz 2023-12-05T23:07:28.807 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64010.log 2023-12-05T23:07:28.807 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43178.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43178.log.gz 2023-12-05T23:07:28.808 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40238.log 2023-12-05T23:07:28.809 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64010.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64010.log.gz 2023-12-05T23:07:28.809 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69952.log 2023-12-05T23:07:28.810 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.40238.log: 73.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40238.log.gz 2023-12-05T23:07:28.815 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80801.log 2023-12-05T23:07:28.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.69952.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69952.log.gz 2023-12-05T23:07:28.815 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65385.log 2023-12-05T23:07:28.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80801.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80801.log.gz 2023-12-05T23:07:28.816 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69526.log 2023-12-05T23:07:28.816 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65385.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65385.log.gz 2023-12-05T23:07:28.816 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46730.log 2023-12-05T23:07:28.816 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69526.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69526.log.gz 2023-12-05T23:07:28.816 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80125.log 2023-12-05T23:07:28.817 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46730.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46730.log.gz 2023-12-05T23:07:28.817 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61176.log 2023-12-05T23:07:28.817 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80125.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80125.log.gz 2023-12-05T23:07:28.817 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49561.log 2023-12-05T23:07:28.818 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61176.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61176.log.gz 2023-12-05T23:07:28.818 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76262.log 2023-12-05T23:07:28.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49561.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49561.log.gz 2023-12-05T23:07:28.820 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57408.log 2023-12-05T23:07:28.822 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76262.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76262.log.gz 2023-12-05T23:07:28.823 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69841.log 2023-12-05T23:07:28.825 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57408.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57408.log.gz 2023-12-05T23:07:28.825 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78825.log 2023-12-05T23:07:28.826 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.69841.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69841.log.gz 2023-12-05T23:07:28.826 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82023.log 2023-12-05T23:07:28.827 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78825.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78825.log.gz 2023-12-05T23:07:28.827 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67699.log 2023-12-05T23:07:28.828 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82023.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82023.log.gz 2023-12-05T23:07:28.828 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42740.log 2023-12-05T23:07:28.829 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67699.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67699.log.gz 2023-12-05T23:07:28.829 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70402.log 2023-12-05T23:07:28.836 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42740.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42740.log.gz 2023-12-05T23:07:28.836 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40081.log 2023-12-05T23:07:28.837 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70402.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70402.log.gz 2023-12-05T23:07:28.837 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79709.log 2023-12-05T23:07:28.838 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40081.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40081.log.gz 2023-12-05T23:07:28.838 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55368.log 2023-12-05T23:07:28.839 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79709.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79709.log.gz 2023-12-05T23:07:28.839 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52430.log 2023-12-05T23:07:28.840 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55368.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55368.log.gz 2023-12-05T23:07:28.841 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53874.log 2023-12-05T23:07:28.841 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52430.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52430.log.gz 2023-12-05T23:07:28.842 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78044.log 2023-12-05T23:07:28.843 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53874.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53874.log.gz 2023-12-05T23:07:28.843 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69360.log 2023-12-05T23:07:28.844 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78044.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78044.log.gz 2023-12-05T23:07:28.844 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54593.log 2023-12-05T23:07:28.847 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69360.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69360.log.gz 2023-12-05T23:07:28.847 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53897.log 2023-12-05T23:07:28.848 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54593.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54593.log.gz 2023-12-05T23:07:28.848 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62614.log 2023-12-05T23:07:28.849 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53897.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53897.log.gz 2023-12-05T23:07:28.850 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78278.log 2023-12-05T23:07:28.850 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62614.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62614.log.gz 2023-12-05T23:07:28.851 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56931.log 2023-12-05T23:07:28.851 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78278.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78278.log.gz 2023-12-05T23:07:28.852 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66193.log 2023-12-05T23:07:28.854 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56931.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56931.log.gz 2023-12-05T23:07:28.854 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49292.log 2023-12-05T23:07:28.855 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66193.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66193.log.gz 2023-12-05T23:07:28.855 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68525.log 2023-12-05T23:07:28.856 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49292.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49292.log.gz 2023-12-05T23:07:28.857 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72090.log 2023-12-05T23:07:28.860 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68525.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68525.log.gz 2023-12-05T23:07:28.860 INFO:teuthology.orchestra.run.smithi190.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.49605.log.gz 2023-12-05T23:07:28.860 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57534.log 2023-12-05T23:07:28.860 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.72090.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72090.log.gz 2023-12-05T23:07:28.861 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43019.log 2023-12-05T23:07:28.861 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63300.log 2023-12-05T23:07:28.862 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57534.log: /var/log/ceph/cluster2-client.mirror.43019.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57534.log.gz 2023-12-05T23:07:28.862 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71032.log 2023-12-05T23:07:28.863 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63300.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63300.log.gz 2023-12-05T23:07:28.863 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40338.log 2023-12-05T23:07:28.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71032.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71032.log.gz 2023-12-05T23:07:28.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65518.log 2023-12-05T23:07:28.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40338.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40338.log.gz 2023-12-05T23:07:28.865 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67016.log 2023-12-05T23:07:28.867 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65518.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65518.log.gz 2023-12-05T23:07:28.867 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77576.log 2023-12-05T23:07:28.868 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67016.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67016.log.gz 2023-12-05T23:07:28.868 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40056.log 2023-12-05T23:07:28.869 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77576.log.gz 2023-12-05T23:07:28.869 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73411.log 2023-12-05T23:07:28.870 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40056.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40056.log.gz 2023-12-05T23:07:28.870 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42720.log 2023-12-05T23:07:28.871 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73411.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73411.log.gz 2023-12-05T23:07:28.871 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67222.log 2023-12-05T23:07:28.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42720.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42720.log.gz 2023-12-05T23:07:28.877 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80827.log 2023-12-05T23:07:28.880 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67222.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67222.log.gz 2023-12-05T23:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67036.log 2023-12-05T23:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43019.log.gz 2023-12-05T23:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80827.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80827.log.gz 2023-12-05T23:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67657.log 2023-12-05T23:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69381.log 2023-12-05T23:07:28.884 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67036.log: /var/log/ceph/cluster2-client.mirror.67657.log: 90.4% 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67036.log.gz 2023-12-05T23:07:28.884 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67657.log.gz 2023-12-05T23:07:28.885 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65091.log 2023-12-05T23:07:28.885 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65172.log 2023-12-05T23:07:28.886 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69381.log: /var/log/ceph/cluster2-client.mirror.65091.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69381.log.gz 2023-12-05T23:07:28.886 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65091.log.gz 2023-12-05T23:07:28.886 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65454.log 2023-12-05T23:07:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65172.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65172.log.gz 2023-12-05T23:07:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61574.log 2023-12-05T23:07:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69443.log 2023-12-05T23:07:28.888 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65454.log: /var/log/ceph/cluster2-client.mirror.61574.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65454.log.gz 2023-12-05T23:07:28.888 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42042.log 2023-12-05T23:07:28.889 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61574.log.gz 2023-12-05T23:07:28.889 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69443.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69443.log.gz 2023-12-05T23:07:28.890 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70695.log 2023-12-05T23:07:28.891 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52684.log 2023-12-05T23:07:28.891 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.42042.log: /var/log/ceph/cluster1-client.mirror.70695.log: 86.9% 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.42042.log.gz 2023-12-05T23:07:28.891 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70695.log.gz 2023-12-05T23:07:28.891 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74404.log 2023-12-05T23:07:28.892 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47082.log 2023-12-05T23:07:28.892 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52684.log: /var/log/ceph/cluster2-client.mirror.74404.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52684.log.gz 2023-12-05T23:07:28.893 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58295.log 2023-12-05T23:07:28.893 INFO:teuthology.orchestra.run.smithi190.stderr: 91.4%/var/log/ceph/cluster2-client.mirror.47082.log: -- replaced with /var/log/ceph/cluster2-client.mirror.74404.log.gz 2023-12-05T23:07:28.894 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73833.log.gz 2023-12-05T23:07:28.898 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47082.log.gz 2023-12-05T23:07:28.899 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53497.log 2023-12-05T23:07:28.899 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63917.log 2023-12-05T23:07:28.899 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56301.log 2023-12-05T23:07:28.900 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63917.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53497.log.gz 2023-12-05T23:07:28.901 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63917.log.gz 2023-12-05T23:07:28.901 INFO:teuthology.orchestra.run.smithi190.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58295.log.gz 2023-12-05T23:07:28.901 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64351.log 2023-12-05T23:07:28.901 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53120.log 2023-12-05T23:07:28.901 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46427.log 2023-12-05T23:07:28.903 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53120.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56301.log.gz 2023-12-05T23:07:28.903 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64351.log.gz 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53120.log.gz 2023-12-05T23:07:28.903 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:28.903 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44940.log 2023-12-05T23:07:28.903 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44962.log 2023-12-05T23:07:28.903 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.46427.log: /var/log/ceph/cluster2-client.mirror.44940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78435.log 2023-12-05T23:07:28.905 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44962.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.46427.log.gz 2023-12-05T23:07:28.905 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44962.log.gz 2023-12-05T23:07:28.905 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60805.log 2023-12-05T23:07:28.905 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78435.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44940.log.gz 2023-12-05T23:07:28.905 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78435.log.gz 2023-12-05T23:07:28.905 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49133.log 2023-12-05T23:07:28.906 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61790.log 2023-12-05T23:07:28.906 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77082.log 2023-12-05T23:07:28.906 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61790.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60805.log.gz 2023-12-05T23:07:28.907 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53920.log 2023-12-05T23:07:28.907 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61790.log.gz 2023-12-05T23:07:28.907 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77082.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77082.log.gz 2023-12-05T23:07:28.907 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68691.log 2023-12-05T23:07:28.908 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82335.log 2023-12-05T23:07:28.909 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68691.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53920.log.gz 2023-12-05T23:07:28.909 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68691.log.gz 2023-12-05T23:07:28.909 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75305.log 2023-12-05T23:07:28.909 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66892.log 2023-12-05T23:07:28.909 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82335.log.gz 2023-12-05T23:07:28.909 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75305.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75305.log.gz 2023-12-05T23:07:28.910 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40376.log 2023-12-05T23:07:28.910 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61511.log 2023-12-05T23:07:28.911 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66892.log: /var/log/ceph/cluster1-client.mirror.40376.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66892.log.gz 2023-12-05T23:07:28.911 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61217.log 2023-12-05T23:07:28.912 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61511.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61511.log.gz 2023-12-05T23:07:28.913 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63006.log 2023-12-05T23:07:28.913 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61217.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40376.log.gz 2023-12-05T23:07:28.914 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72149.log 2023-12-05T23:07:28.914 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63006.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63006.log.gz 2023-12-05T23:07:28.915 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50610.log 2023-12-05T23:07:28.915 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.72149.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61217.log.gz 2023-12-05T23:07:28.916 INFO:teuthology.orchestra.run.smithi190.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72149.log.gz 2023-12-05T23:07:28.916 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52605.log 2023-12-05T23:07:28.916 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50610.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50610.log.gz 2023-12-05T23:07:28.917 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82231.log 2023-12-05T23:07:28.917 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76156.log 2023-12-05T23:07:28.918 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52605.log: /var/log/ceph/cluster2-client.admin.82231.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82231.log.gz 2023-12-05T23:07:28.919 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77212.log 2023-12-05T23:07:28.919 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76156.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52605.log.gz 2023-12-05T23:07:28.920 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69546.log 2023-12-05T23:07:28.920 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77212.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77212.log.gz 2023-12-05T23:07:28.920 INFO:teuthology.orchestra.run.smithi190.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76156.log.gz 2023-12-05T23:07:28.920 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56476.log 2023-12-05T23:07:28.921 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67802.log 2023-12-05T23:07:28.921 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69546.log: /var/log/ceph/cluster2-client.mirror.56476.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56476.log.gz 2023-12-05T23:07:28.922 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76004.log 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67802.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67802.log.gz 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69546.log.gz 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80879.log 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77420.log 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76004.log.gz 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80879.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40790.log 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.80879.log.gz 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77420.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77420.log.gz 2023-12-05T23:07:28.925 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44244.log 2023-12-05T23:07:28.926 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.40790.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.40790.log.gz 2023-12-05T23:07:28.926 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68153.log 2023-12-05T23:07:28.926 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79033.log 2023-12-05T23:07:28.929 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44244.log: /var/log/ceph/cluster2-client.mirror.68153.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68153.log.gz 2023-12-05T23:07:28.931 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44244.log.gz 2023-12-05T23:07:28.932 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80385.log 2023-12-05T23:07:28.933 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79033.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50494.log 2023-12-05T23:07:28.933 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79033.log.gz 2023-12-05T23:07:28.933 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66995.log 2023-12-05T23:07:28.933 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80385.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80385.log.gz 2023-12-05T23:07:28.948 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50494.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49694.log 2023-12-05T23:07:28.949 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66995.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66995.log.gz 2023-12-05T23:07:28.950 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.53164.log 2023-12-05T23:07:28.950 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49694.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49694.log.gz 2023-12-05T23:07:28.951 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71707.log 2023-12-05T23:07:28.960 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.2.53164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52362.log 2023-12-05T23:07:28.961 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71707.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71707.log.gz 2023-12-05T23:07:28.968 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50822.log 2023-12-05T23:07:28.968 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74213.log.gz 2023-12-05T23:07:28.969 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52362.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52362.log.gz 2023-12-05T23:07:28.970 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70087.log 2023-12-05T23:07:28.992 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60783.log 2023-12-05T23:07:28.993 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70087.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70087.log.gz 2023-12-05T23:07:29.004 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74966.log 2023-12-05T23:07:29.006 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60783.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60783.log.gz 2023-12-05T23:07:29.007 INFO:teuthology.orchestra.run.smithi190.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.53164.log.gz 2023-12-05T23:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81763.log 2023-12-05T23:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74966.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74966.log.gz 2023-12-05T23:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44461.log 2023-12-05T23:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81763.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81763.log.gz 2023-12-05T23:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76677.log 2023-12-05T23:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.44461.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.44461.log.gz 2023-12-05T23:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66830.log 2023-12-05T23:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76677.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76677.log.gz 2023-12-05T23:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51079.log 2023-12-05T23:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66830.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66830.log.gz 2023-12-05T23:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62456.log 2023-12-05T23:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51079.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51908.log.gz 2023-12-05T23:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77654.log 2023-12-05T23:07:29.038 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62456.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62456.log.gz 2023-12-05T23:07:29.039 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56108.log 2023-12-05T23:07:29.039 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77654.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77654.log.gz 2023-12-05T23:07:29.040 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70740.log 2023-12-05T23:07:29.040 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56108.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56108.log.gz 2023-12-05T23:07:29.041 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82179.log 2023-12-05T23:07:29.042 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70740.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70740.log.gz 2023-12-05T23:07:29.042 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46289.log 2023-12-05T23:07:29.043 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82179.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82179.log.gz 2023-12-05T23:07:29.043 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64246.log 2023-12-05T23:07:29.049 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46289.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46289.log.gz 2023-12-05T23:07:29.049 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60936.log 2023-12-05T23:07:29.050 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64246.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64246.log.gz 2023-12-05T23:07:29.050 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46618.log 2023-12-05T23:07:29.051 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60936.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60936.log.gz 2023-12-05T23:07:29.051 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39981.log 2023-12-05T23:07:29.053 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46618.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46618.log.gz 2023-12-05T23:07:29.053 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44561.log 2023-12-05T23:07:29.054 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.39981.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39981.log.gz 2023-12-05T23:07:29.054 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67078.log 2023-12-05T23:07:29.056 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44561.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44561.log.gz 2023-12-05T23:07:29.056 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62660.log 2023-12-05T23:07:29.057 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67078.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67078.log.gz 2023-12-05T23:07:29.057 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58379.log 2023-12-05T23:07:29.058 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62660.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62660.log.gz 2023-12-05T23:07:29.058 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58359.log 2023-12-05T23:07:29.059 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58379.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58379.log.gz 2023-12-05T23:07:29.059 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41131.log 2023-12-05T23:07:29.061 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58359.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58359.log.gz 2023-12-05T23:07:29.061 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58653.log 2023-12-05T23:07:29.062 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41131.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41131.log.gz 2023-12-05T23:07:29.062 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71820.log 2023-12-05T23:07:29.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58653.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58653.log.gz 2023-12-05T23:07:29.063 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69629.log 2023-12-05T23:07:29.064 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71820.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71820.log.gz 2023-12-05T23:07:29.064 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44650.log 2023-12-05T23:07:29.065 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69629.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69629.log.gz 2023-12-05T23:07:29.066 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60326.log 2023-12-05T23:07:29.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44650.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44650.log.gz 2023-12-05T23:07:29.067 INFO:teuthology.orchestra.run.smithi190.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-12-05T23:07:29.067 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56413.log 2023-12-05T23:07:29.067 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81867.log 2023-12-05T23:07:29.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60326.log: /var/log/ceph/cluster1-client.mirror.56413.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56413.log.gz 2023-12-05T23:07:29.068 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.44962.log 2023-12-05T23:07:29.069 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81867.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81867.log.gz 2023-12-05T23:07:29.070 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60127.log 2023-12-05T23:07:29.070 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60326.log.gz/var/log/ceph/cluster2-client.mirror.1.44962.log: 2023-12-05T23:07:29.070 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74191.log 2023-12-05T23:07:29.071 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60127.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60127.log.gz 2023-12-05T23:07:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81347.log 2023-12-05T23:07:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74191.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74191.log.gz 2023-12-05T23:07:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80567.log 2023-12-05T23:07:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81347.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81347.log.gz 2023-12-05T23:07:29.074 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73691.log 2023-12-05T23:07:29.074 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80567.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80567.log.gz 2023-12-05T23:07:29.075 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55152.log 2023-12-05T23:07:29.075 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73691.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73691.log.gz 2023-12-05T23:07:29.076 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64054.log 2023-12-05T23:07:29.077 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55152.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55152.log.gz 2023-12-05T23:07:29.077 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51402.log 2023-12-05T23:07:29.077 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64054.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.44962.log.gz 2023-12-05T23:07:29.078 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73712.log 2023-12-05T23:07:29.079 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51402.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64054.log.gz 2023-12-05T23:07:29.080 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70155.log 2023-12-05T23:07:29.081 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73712.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73712.log.gz 2023-12-05T23:07:29.081 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54261.log 2023-12-05T23:07:29.082 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70155.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70155.log.gz 2023-12-05T23:07:29.082 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46352.log 2023-12-05T23:07:29.083 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54261.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54261.log.gz 2023-12-05T23:07:29.083 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82517.log 2023-12-05T23:07:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46352.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46352.log.gz 2023-12-05T23:07:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69650.log 2023-12-05T23:07:29.090 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82517.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82517.log.gz 2023-12-05T23:07:29.090 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42376.log 2023-12-05T23:07:29.091 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69650.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69650.log.gz 2023-12-05T23:07:29.091 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66547.log 2023-12-05T23:07:29.097 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42376.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42376.log.gz 2023-12-05T23:07:29.098 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77264.log 2023-12-05T23:07:29.099 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66547.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66547.log.gz 2023-12-05T23:07:29.099 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66643.log 2023-12-05T23:07:29.100 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77264.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77264.log.gz 2023-12-05T23:07:29.100 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71010.log 2023-12-05T23:07:29.101 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66643.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66643.log.gz 2023-12-05T23:07:29.101 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68567.log 2023-12-05T23:07:29.102 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71010.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71010.log.gz 2023-12-05T23:07:29.102 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61197.log 2023-12-05T23:07:29.103 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68567.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68567.log.gz 2023-12-05T23:07:29.103 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65586.log 2023-12-05T23:07:29.106 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61197.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61197.log.gz 2023-12-05T23:07:29.106 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54570.log 2023-12-05T23:07:29.107 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65586.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65586.log.gz 2023-12-05T23:07:29.108 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60306.log 2023-12-05T23:07:29.109 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54570.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54570.log.gz 2023-12-05T23:07:29.109 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70807.log 2023-12-05T23:07:29.110 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60306.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50494.log.gz 2023-12-05T23:07:29.111 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60306.log.gz 2023-12-05T23:07:29.112 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43998.log 2023-12-05T23:07:29.112 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53428.log 2023-12-05T23:07:29.112 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9%/var/log/ceph/cluster1-client.mirror.43998.log: -- replaced with /var/log/ceph/cluster1-client.mirror.70807.log.gz 2023-12-05T23:07:29.114 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48626.log 2023-12-05T23:07:29.114 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43998.log.gz 2023-12-05T23:07:29.128 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54284.log 2023-12-05T23:07:29.148 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64223.log 2023-12-05T23:07:29.149 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54284.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54284.log.gz 2023-12-05T23:07:29.156 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81945.log 2023-12-05T23:07:29.157 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64223.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64223.log.gz 2023-12-05T23:07:29.173 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47039.log 2023-12-05T23:07:29.173 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81945.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81945.log.gz 2023-12-05T23:07:29.188 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54458.log 2023-12-05T23:07:29.194 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47039.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47039.log.gz 2023-12-05T23:07:29.200 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.64698.log 2023-12-05T23:07:29.201 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54458.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54458.log.gz 2023-12-05T23:07:29.205 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75470.log.gz 2023-12-05T23:07:29.205 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55198.log 2023-12-05T23:07:29.224 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.2.64698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66501.log 2023-12-05T23:07:29.225 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55198.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55198.log.gz 2023-12-05T23:07:29.225 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82361.log 2023-12-05T23:07:29.225 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66501.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66501.log.gz 2023-12-05T23:07:29.234 INFO:teuthology.orchestra.run.smithi190.stderr: 93.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62723.log 2023-12-05T23:07:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82361.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82361.log.gz 2023-12-05T23:07:29.235 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.64698.log.gz 2023-12-05T23:07:29.235 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.75032.log 2023-12-05T23:07:29.236 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62723.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62723.log.gz 2023-12-05T23:07:29.236 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61552.log 2023-12-05T23:07:29.239 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.75032.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.75032.log.gz 2023-12-05T23:07:29.240 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76063.log 2023-12-05T23:07:29.241 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61552.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61552.log.gz 2023-12-05T23:07:29.242 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48351.log 2023-12-05T23:07:29.256 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56042.log 2023-12-05T23:07:29.257 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48351.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48351.log.gz 2023-12-05T23:07:29.272 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67160.log 2023-12-05T23:07:29.273 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56042.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56042.log.gz 2023-12-05T23:07:29.273 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69402.log 2023-12-05T23:07:29.280 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67160.log: 90.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80463.log 2023-12-05T23:07:29.281 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69402.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69402.log.gz 2023-12-05T23:07:29.288 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80723.log 2023-12-05T23:07:29.288 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80463.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80463.log.gz 2023-12-05T23:07:29.292 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67160.log.gz 2023-12-05T23:07:29.296 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41258.log 2023-12-05T23:07:29.297 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80723.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80723.log.gz 2023-12-05T23:07:29.308 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62568.log 2023-12-05T23:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41258.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41258.log.gz 2023-12-05T23:07:29.320 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42120.log 2023-12-05T23:07:29.321 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62568.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62568.log.gz 2023-12-05T23:07:29.332 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79735.log 2023-12-05T23:07:29.333 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.42120.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.42120.log.gz 2023-12-05T23:07:29.336 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57367.log 2023-12-05T23:07:29.336 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79735.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79735.log.gz 2023-12-05T23:07:29.337 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60479.log 2023-12-05T23:07:29.338 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57367.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57367.log.gz 2023-12-05T23:07:29.351 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51079.log.gz 2023-12-05T23:07:29.352 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79813.log 2023-12-05T23:07:29.352 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60479.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60479.log.gz 2023-12-05T23:07:29.353 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65845.log 2023-12-05T23:07:29.353 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79813.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79813.log.gz 2023-12-05T23:07:29.354 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63075.log 2023-12-05T23:07:29.354 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65845.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49133.log.gz 2023-12-05T23:07:29.354 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65845.log.gz 2023-12-05T23:07:29.356 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60018.log 2023-12-05T23:07:29.356 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60696.log 2023-12-05T23:07:29.356 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60018.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63075.log.gz 2023-12-05T23:07:29.357 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60018.log.gz 2023-12-05T23:07:29.357 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71572.log 2023-12-05T23:07:29.358 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45707.log 2023-12-05T23:07:29.358 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60696.log: /var/log/ceph/cluster1-client.mirror.71572.log: 86.8% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60696.log.gz 2023-12-05T23:07:29.358 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71572.log.gz 2023-12-05T23:07:29.358 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75983.log 2023-12-05T23:07:29.359 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57732.log 2023-12-05T23:07:29.359 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45707.log: /var/log/ceph/cluster2-client.mirror.75983.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45707.log.gz 2023-12-05T23:07:29.359 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.40376.log 2023-12-05T23:07:29.359 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51402.log.gz 2023-12-05T23:07:29.360 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57732.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57732.log.gz 2023-12-05T23:07:29.360 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46882.log 2023-12-05T23:07:29.361 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52704.log 2023-12-05T23:07:29.361 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.3.40376.log: /var/log/ceph/cluster2-client.mirror.46882.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75983.log.gz 2023-12-05T23:07:29.361 INFO:teuthology.orchestra.run.smithi190.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46882.log.gz 2023-12-05T23:07:29.376 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46994.log 2023-12-05T23:07:29.379 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63052.log 2023-12-05T23:07:29.379 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52704.log: /var/log/ceph/cluster1-client.mirror.46994.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52704.log.gz 2023-12-05T23:07:29.379 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46994.log.gz 2023-12-05T23:07:29.379 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50822.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65819.log 2023-12-05T23:07:29.380 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:29.380 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50990.log 2023-12-05T23:07:29.380 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42263.log 2023-12-05T23:07:29.380 INFO:teuthology.orchestra.run.smithi190.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63052.log.gz 2023-12-05T23:07:29.380 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50990.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65819.log.gz 2023-12-05T23:07:29.380 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68009.log 2023-12-05T23:07:29.380 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58254.log 2023-12-05T23:07:29.381 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68009.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42263.log.gz 2023-12-05T23:07:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64154.log 2023-12-05T23:07:29.382 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.58254.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68009.log.gz 2023-12-05T23:07:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- 87.7% /var/log/ceph/cluster2-client.mirror.58852.log 2023-12-05T23:07:29.382 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58254.log.gz 2023-12-05T23:07:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50531.log 2023-12-05T23:07:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64154.log: /var/log/ceph/cluster2-client.mirror.58852.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64154.log.gz 2023-12-05T23:07:29.382 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58852.log.gz 2023-12-05T23:07:29.383 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69236.log 2023-12-05T23:07:29.383 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67554.log 2023-12-05T23:07:29.383 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69236.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50531.log.gz 2023-12-05T23:07:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77836.log 2023-12-05T23:07:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67554.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67554.log.gz 2023-12-05T23:07:29.386 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69236.log.gz 2023-12-05T23:07:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58965.log 2023-12-05T23:07:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50155.log 2023-12-05T23:07:29.386 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77836.log.gz 2023-12-05T23:07:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73791.log 2023-12-05T23:07:29.387 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50155.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58965.log.gz 2023-12-05T23:07:29.388 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49731.log 2023-12-05T23:07:29.389 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73791.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73791.log.gz 2023-12-05T23:07:29.390 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52746.log 2023-12-05T23:07:29.390 INFO:teuthology.orchestra.run.smithi190.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50155.log.gz 2023-12-05T23:07:29.390 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68443.log 2023-12-05T23:07:29.391 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52746.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49731.log.gz 2023-12-05T23:07:29.391 INFO:teuthology.orchestra.run.smithi190.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52746.log.gz 2023-12-05T23:07:29.391 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57557.log 2023-12-05T23:07:29.392 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75877.log 2023-12-05T23:07:29.392 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68443.log: /var/log/ceph/cluster1-client.mirror.57557.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68443.log.gz 2023-12-05T23:07:29.392 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74989.log 2023-12-05T23:07:29.392 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57557.log.gz 2023-12-05T23:07:29.393 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76850.log 2023-12-05T23:07:29.393 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75877.log: /var/log/ceph/cluster1-client.mirror.74989.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74989.log.gz 2023-12-05T23:07:29.394 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69885.log 2023-12-05T23:07:29.394 INFO:teuthology.orchestra.run.smithi190.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75877.log.gz 2023-12-05T23:07:29.395 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79917.log 2023-12-05T23:07:29.395 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76850.log: /var/log/ceph/cluster1-client.mirror.69885.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76850.log.gz 2023-12-05T23:07:29.395 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69885.log.gz 2023-12-05T23:07:29.395 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40181.log 2023-12-05T23:07:29.397 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48995.log 2023-12-05T23:07:29.397 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79917.log: /var/log/ceph/cluster2-client.admin.40181.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79917.log.gz 2023-12-05T23:07:29.397 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40181.log.gz 2023-12-05T23:07:29.397 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48809.log 2023-12-05T23:07:29.397 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56841.log 2023-12-05T23:07:29.398 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48995.log: /var/log/ceph/cluster2-client.mirror.48809.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48995.log.gz 2023-12-05T23:07:29.398 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48809.log.gz 2023-12-05T23:07:29.398 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76921.log 2023-12-05T23:07:29.399 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39904.log 2023-12-05T23:07:29.399 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56841.log: /var/log/ceph/cluster1-client.admin.76921.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76921.log.gz 2023-12-05T23:07:29.399 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56841.log.gz 2023-12-05T23:07:29.399 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66324.log 2023-12-05T23:07:29.399 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39670.log 2023-12-05T23:07:29.399 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39904.log.gz 2023-12-05T23:07:29.400 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64200.log 2023-12-05T23:07:29.400 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66324.log: /var/log/ceph/cluster2-client.admin.39670.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39670.log.gz 2023-12-05T23:07:29.400 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.46018.log 2023-12-05T23:07:29.401 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64200.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64200.log.gz 2023-12-05T23:07:29.401 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68968.log 2023-12-05T23:07:29.406 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.2.46018.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66324.log.gz 2023-12-05T23:07:29.406 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45817.log 2023-12-05T23:07:29.407 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68968.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68968.log.gz 2023-12-05T23:07:29.407 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48519.log 2023-12-05T23:07:29.411 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45817.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45817.log.gz 2023-12-05T23:07:29.416 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.46018.log.gz 2023-12-05T23:07:29.416 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64440.log 2023-12-05T23:07:29.418 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41221.log 2023-12-05T23:07:29.419 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64440.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48519.log.gz 2023-12-05T23:07:29.419 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64440.log.gz 2023-12-05T23:07:29.419 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59906.log 2023-12-05T23:07:29.419 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67119.log 2023-12-05T23:07:29.419 INFO:teuthology.orchestra.run.smithi190.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41221.log.gz 2023-12-05T23:07:29.420 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59906.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59906.log.gz 2023-12-05T23:07:29.420 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75448.log 2023-12-05T23:07:29.420 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42220.log 2023-12-05T23:07:29.421 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67119.log: /var/log/ceph/cluster2-client.mirror.75448.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75448.log.gz 2023-12-05T23:07:29.421 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67119.log.gz 2023-12-05T23:07:29.421 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55052.log 2023-12-05T23:07:29.422 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50385.log 2023-12-05T23:07:29.422 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42220.log: /var/log/ceph/cluster2-client.mirror.55052.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42220.log.gz 2023-12-05T23:07:29.423 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55760.log 2023-12-05T23:07:29.427 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50385.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55052.log.gz 2023-12-05T23:07:29.427 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50385.log.gz 2023-12-05T23:07:29.427 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63161.log 2023-12-05T23:07:29.427 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51237.log 2023-12-05T23:07:29.427 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55760.log: /var/log/ceph/cluster2-client.mirror.63161.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63161.log.gz 2023-12-05T23:07:29.427 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55760.log.gz 2023-12-05T23:07:29.427 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45904.log 2023-12-05T23:07:29.428 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46332.log 2023-12-05T23:07:29.428 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51237.log: /var/log/ceph/cluster2-client.mirror.45904.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.40376.log.gz 2023-12-05T23:07:29.441 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50365.log 2023-12-05T23:07:29.446 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46332.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46332.log.gz 2023-12-05T23:07:29.446 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80957.log 2023-12-05T23:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50365.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50365.log.gz 2023-12-05T23:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55244.log 2023-12-05T23:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80957.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80957.log.gz 2023-12-05T23:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82908.log 2023-12-05T23:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55244.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55244.log.gz 2023-12-05T23:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70110.log 2023-12-05T23:07:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82908.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82908.log.gz 2023-12-05T23:07:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76135.log 2023-12-05T23:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70110.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70110.log.gz 2023-12-05T23:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83038.log 2023-12-05T23:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76135.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76135.log.gz 2023-12-05T23:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61351.log 2023-12-05T23:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83038.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83038.log.gz 2023-12-05T23:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49012.log 2023-12-05T23:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61351.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61351.log.gz 2023-12-05T23:07:29.456 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57077.log 2023-12-05T23:07:29.457 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49012.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49012.log.gz 2023-12-05T23:07:29.457 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60847.log 2023-12-05T23:07:29.458 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57077.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57077.log.gz 2023-12-05T23:07:29.459 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49484.log 2023-12-05T23:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60847.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60847.log.gz 2023-12-05T23:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55739.log 2023-12-05T23:07:29.477 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42997.log 2023-12-05T23:07:29.478 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55739.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55739.log.gz 2023-12-05T23:07:29.492 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72186.log 2023-12-05T23:07:29.494 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42997.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42997.log.gz 2023-12-05T23:07:29.494 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43640.log 2023-12-05T23:07:29.495 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.72186.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72186.log.gz 2023-12-05T23:07:29.512 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74622.log 2023-12-05T23:07:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43640.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43640.log.gz 2023-12-05T23:07:29.520 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55524.log 2023-12-05T23:07:29.520 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74622.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74622.log.gz 2023-12-05T23:07:29.528 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34886.log 2023-12-05T23:07:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55524.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55524.log.gz 2023-12-05T23:07:29.536 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74910.log 2023-12-05T23:07:29.536 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.34886.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34886.log.gz 2023-12-05T23:07:29.542 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53428.log.gz 2023-12-05T23:07:29.543 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80229.log 2023-12-05T23:07:29.544 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74910.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74910.log.gz 2023-12-05T23:07:29.544 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47102.log 2023-12-05T23:07:29.544 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80229.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80229.log.gz 2023-12-05T23:07:29.545 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70605.log 2023-12-05T23:07:29.550 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47102.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47102.log.gz 2023-12-05T23:07:29.551 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43291.log 2023-12-05T23:07:29.551 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70605.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70605.log.gz 2023-12-05T23:07:29.552 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47059.log 2023-12-05T23:07:29.557 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43291.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76063.log.gz 2023-12-05T23:07:29.558 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43291.log.gz 2023-12-05T23:07:29.561 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62175.log 2023-12-05T23:07:29.561 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41148.log 2023-12-05T23:07:29.561 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62175.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50990.log.gz 2023-12-05T23:07:29.561 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62175.log.gz 2023-12-05T23:07:29.561 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44780.log 2023-12-05T23:07:29.562 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66046.log 2023-12-05T23:07:29.562 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41148.log: /var/log/ceph/cluster1-client.mirror.44780.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41148.log.gz 2023-12-05T23:07:29.562 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66127.log 2023-12-05T23:07:29.564 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66046.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66046.log.gz 2023-12-05T23:07:29.564 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51928.log 2023-12-05T23:07:29.564 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47059.log.gz/var/log/ceph/cluster2-client.mirror.66127.log: 2023-12-05T23:07:29.569 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66127.log.gz 2023-12-05T23:07:29.569 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52862.log 2023-12-05T23:07:29.569 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68029.log 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51928.log: /var/log/ceph/cluster2-client.mirror.52862.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51928.log.gz 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44780.log.gz 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52862.log.gz 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62591.log 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65799.log 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68029.log: /var/log/ceph/cluster1-client.mirror.62591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48604.log 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65799.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62591.log.gz 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65799.log.gz 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68029.log.gz 2023-12-05T23:07:29.570 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43813.log 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49990.log 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48604.log: /var/log/ceph/cluster2-client.mirror.43813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78565.log 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr: 74.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43813.log.gz 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49990.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48604.log.gz 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74639.log 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67243.log 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78565.log: /var/log/ceph/cluster2-client.mirror.74639.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78565.log.gz 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44886.log 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67243.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74639.log.gz 2023-12-05T23:07:29.573 INFO:teuthology.orchestra.run.smithi190.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67243.log.gz 2023-12-05T23:07:29.574 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46199.log 2023-12-05T23:07:29.574 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68608.log 2023-12-05T23:07:29.575 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.44886.log: /var/log/ceph/cluster1-client.mirror.46199.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44886.log.gz 2023-12-05T23:07:29.575 INFO:teuthology.orchestra.run.smithi190.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46199.log.gz 2023-12-05T23:07:29.575 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79293.log 2023-12-05T23:07:29.577 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57601.log 2023-12-05T23:07:29.577 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68608.log: /var/log/ceph/cluster2-client.admin.79293.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79293.log.gz 2023-12-05T23:07:29.577 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68608.log.gz 2023-12-05T23:07:29.578 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44267.log 2023-12-05T23:07:29.578 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79241.log 2023-12-05T23:07:29.580 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57601.log: /var/log/ceph/cluster2-client.mirror.44267.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57601.log.gz 2023-12-05T23:07:29.581 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39387.log 2023-12-05T23:07:29.581 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79241.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79241.log.gz 2023-12-05T23:07:29.581 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.61921.log 2023-12-05T23:07:29.581 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39387.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39387.log.gz 2023-12-05T23:07:29.581 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42973.log 2023-12-05T23:07:29.582 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.61921.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44267.log.gz 2023-12-05T23:07:29.591 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48626.log.gz 2023-12-05T23:07:29.592 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54985.log 2023-12-05T23:07:29.592 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70470.log 2023-12-05T23:07:29.593 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54985.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42973.log.gz 2023-12-05T23:07:29.593 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54985.log.gz 2023-12-05T23:07:29.593 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77732.log 2023-12-05T23:07:29.594 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52842.log 2023-12-05T23:07:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70470.log: /var/log/ceph/cluster2-client.admin.77732.log: 0.0% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70470.log.gz 2023-12-05T23:07:29.595 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.77732.log.gz 2023-12-05T23:07:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52884.log 2023-12-05T23:07:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47618.log 2023-12-05T23:07:29.595 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52842.log: /var/log/ceph/cluster2-client.mirror.52884.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52842.log.gz 2023-12-05T23:07:29.596 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79111.log 2023-12-05T23:07:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47618.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47618.log.gz 2023-12-05T23:07:29.602 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54614.log 2023-12-05T23:07:29.602 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79111.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79111.log.gz 2023-12-05T23:07:29.603 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57388.log 2023-12-05T23:07:29.605 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54614.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54614.log.gz 2023-12-05T23:07:29.605 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55175.log 2023-12-05T23:07:29.608 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57388.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57388.log.gz 2023-12-05T23:07:29.608 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67595.log 2023-12-05T23:07:29.608 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55175.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55175.log.gz 2023-12-05T23:07:29.609 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59884.log 2023-12-05T23:07:29.610 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67595.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.61921.log.gz 2023-12-05T23:07:29.611 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66747.log 2023-12-05T23:07:29.611 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59884.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67595.log.gz 2023-12-05T23:07:29.614 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75032.log 2023-12-05T23:07:29.614 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66747.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59884.log.gz 2023-12-05T23:07:29.614 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47188.log 2023-12-05T23:07:29.615 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66747.log.gz 2023-12-05T23:07:29.615 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75032.log: 85.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75032.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52568.log 2023-12-05T23:07:29.615 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:07:29.615 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75282.log 2023-12-05T23:07:29.619 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52568.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47188.log.gz 2023-12-05T23:07:29.620 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60433.log 2023-12-05T23:07:29.621 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75282.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75282.log.gz 2023-12-05T23:07:29.621 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49524.log 2023-12-05T23:07:29.622 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60433.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60433.log.gz 2023-12-05T23:07:29.622 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54547.log 2023-12-05T23:07:29.624 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49524.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49524.log.gz 2023-12-05T23:07:29.624 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78799.log 2023-12-05T23:07:29.624 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54547.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54547.log.gz 2023-12-05T23:07:29.624 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78226.log 2023-12-05T23:07:29.629 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78799.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78799.log.gz 2023-12-05T23:07:29.629 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74932.log 2023-12-05T23:07:29.630 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78226.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78226.log.gz 2023-12-05T23:07:29.630 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51598.log 2023-12-05T23:07:29.640 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80593.log 2023-12-05T23:07:29.641 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.51598.log: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51598.log.gz 2023-12-05T23:07:29.664 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48222.log 2023-12-05T23:07:29.665 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80593.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80593.log.gz 2023-12-05T23:07:29.676 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63672.log 2023-12-05T23:07:29.677 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48222.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49484.log.gz 2023-12-05T23:07:29.696 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47856.log 2023-12-05T23:07:29.697 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63672.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63672.log.gz 2023-12-05T23:07:29.712 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59328.log 2023-12-05T23:07:29.718 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47856.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47856.log.gz 2023-12-05T23:07:29.719 INFO:teuthology.orchestra.run.smithi190.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48222.log.gz 2023-12-05T23:07:29.719 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58139.log 2023-12-05T23:07:29.721 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59328.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59328.log.gz 2023-12-05T23:07:29.722 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66596.log 2023-12-05T23:07:29.722 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58139.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58139.log.gz 2023-12-05T23:07:29.723 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49911.log 2023-12-05T23:07:29.725 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66596.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66596.log.gz 2023-12-05T23:07:29.725 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70515.log 2023-12-05T23:07:29.740 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73668.log 2023-12-05T23:07:29.741 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70515.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70515.log.gz 2023-12-05T23:07:29.770 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66147.log 2023-12-05T23:07:29.770 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73668.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73668.log.gz 2023-12-05T23:07:29.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44180.log 2023-12-05T23:07:29.777 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66147.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66147.log.gz 2023-12-05T23:07:29.784 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77784.log 2023-12-05T23:07:29.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44180.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44180.log.gz 2023-12-05T23:07:29.792 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58035.log 2023-12-05T23:07:29.792 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77784.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77784.log.gz 2023-12-05T23:07:29.800 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80307.log 2023-12-05T23:07:29.802 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58035.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58035.log.gz 2023-12-05T23:07:29.802 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45904.log.gz 2023-12-05T23:07:29.805 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52588.log 2023-12-05T23:07:29.805 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80307.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80307.log.gz 2023-12-05T23:07:29.805 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.43019.log 2023-12-05T23:07:29.805 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52588.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52588.log.gz 2023-12-05T23:07:29.806 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53724.log 2023-12-05T23:07:29.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.1.43019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74383.log 2023-12-05T23:07:29.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53724.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53724.log.gz 2023-12-05T23:07:29.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82934.log 2023-12-05T23:07:29.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74383.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74383.log.gz 2023-12-05T23:07:29.838 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81295.log 2023-12-05T23:07:29.838 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82934.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82934.log.gz 2023-12-05T23:07:29.856 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78487.log 2023-12-05T23:07:29.857 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81295.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81295.log.gz 2023-12-05T23:07:29.857 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66239.log 2023-12-05T23:07:29.857 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78487.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78487.log.gz 2023-12-05T23:07:29.862 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51237.log.gz 2023-12-05T23:07:29.862 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70762.log 2023-12-05T23:07:29.863 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66239.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66239.log.gz 2023-12-05T23:07:29.863 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66912.log 2023-12-05T23:07:29.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70762.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70762.log.gz 2023-12-05T23:07:29.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68050.log 2023-12-05T23:07:29.867 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66912.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66912.log.gz 2023-12-05T23:07:29.867 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46817.log 2023-12-05T23:07:29.868 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68050.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68050.log.gz 2023-12-05T23:07:29.869 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81217.log 2023-12-05T23:07:29.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46817.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46817.log.gz 2023-12-05T23:07:29.876 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76875.log 2023-12-05T23:07:29.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81217.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81217.log.gz 2023-12-05T23:07:29.877 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55267.log 2023-12-05T23:07:29.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76875.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76875.log.gz 2023-12-05T23:07:29.877 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70357.log 2023-12-05T23:07:29.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55267.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55267.log.gz 2023-12-05T23:07:29.878 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40031.log 2023-12-05T23:07:29.878 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70357.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70357.log.gz 2023-12-05T23:07:29.878 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67719.log 2023-12-05T23:07:29.879 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40031.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40031.log.gz 2023-12-05T23:07:29.880 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69505.log 2023-12-05T23:07:29.882 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67719.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67719.log.gz 2023-12-05T23:07:29.882 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81841.log 2023-12-05T23:07:29.884 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69505.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69505.log.gz 2023-12-05T23:07:29.884 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48371.log 2023-12-05T23:07:29.884 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.43019.log.gz 2023-12-05T23:07:29.884 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81841.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81841.log.gz 2023-12-05T23:07:29.885 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63940.log 2023-12-05T23:07:29.885 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40131.log 2023-12-05T23:07:29.886 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63940.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48371.log.gz 2023-12-05T23:07:29.886 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63940.log.gz 2023-12-05T23:07:29.886 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67140.log 2023-12-05T23:07:29.887 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.40131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68360.log 2023-12-05T23:07:29.887 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40131.log.gz 2023-12-05T23:07:29.888 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59972.log 2023-12-05T23:07:29.888 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68360.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67140.log.gz 2023-12-05T23:07:29.889 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45882.log 2023-12-05T23:07:29.889 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59972.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68360.log.gz 2023-12-05T23:07:29.889 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59972.log.gz 2023-12-05T23:07:29.889 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54873.log 2023-12-05T23:07:29.890 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44581.log 2023-12-05T23:07:29.891 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45882.log: /var/log/ceph/cluster2-client.mirror.54873.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45882.log.gz 2023-12-05T23:07:29.891 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54873.log.gz 2023-12-05T23:07:29.891 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40568.log 2023-12-05T23:07:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78669.log 2023-12-05T23:07:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44581.log: /var/log/ceph/cluster1-client.admin.40568.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.40568.log.gz 2023-12-05T23:07:29.893 INFO:teuthology.orchestra.run.smithi190.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44581.log.gz 2023-12-05T23:07:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55411.log 2023-12-05T23:07:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53983.log 2023-12-05T23:07:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78669.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78669.log.gz 2023-12-05T23:07:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59576.log 2023-12-05T23:07:29.894 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55411.log.gz 2023-12-05T23:07:29.894 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53983.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53983.log.gz 2023-12-05T23:07:29.895 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69862.log 2023-12-05T23:07:29.895 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42355.log 2023-12-05T23:07:29.895 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59576.log: /var/log/ceph/cluster1-client.mirror.69862.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69862.log.gz 2023-12-05T23:07:29.896 INFO:teuthology.orchestra.run.smithi190.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59576.log.gz 2023-12-05T23:07:29.897 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80047.log 2023-12-05T23:07:29.898 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74426.log 2023-12-05T23:07:29.898 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42355.log: /var/log/ceph/cluster2-client.admin.80047.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80047.log.gz 2023-12-05T23:07:29.898 INFO:teuthology.orchestra.run.smithi190.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42355.log.gz 2023-12-05T23:07:29.899 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67988.log 2023-12-05T23:07:29.899 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49852.log 2023-12-05T23:07:29.900 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74426.log: /var/log/ceph/cluster2-client.mirror.67988.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74426.log.gz 2023-12-05T23:07:29.900 INFO:teuthology.orchestra.run.smithi190.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67988.log.gz 2023-12-05T23:07:29.900 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49753.log 2023-12-05T23:07:29.901 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48789.log 2023-12-05T23:07:29.901 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49852.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49852.log.gz 2023-12-05T23:07:29.901 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72208.log 2023-12-05T23:07:29.902 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48789.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48789.log.gz 2023-12-05T23:07:29.903 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57123.log 2023-12-05T23:07:29.915 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.72208.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49911.log.gz 2023-12-05T23:07:29.915 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65241.log 2023-12-05T23:07:29.916 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57123.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57123.log.gz 2023-12-05T23:07:29.916 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66685.log 2023-12-05T23:07:29.917 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65241.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65241.log.gz 2023-12-05T23:07:29.917 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46268.log 2023-12-05T23:07:29.918 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66685.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66685.log.gz 2023-12-05T23:07:29.919 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69092.log 2023-12-05T23:07:29.919 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46268.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46268.log.gz 2023-12-05T23:07:29.920 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57232.log 2023-12-05T23:07:29.921 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69092.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69092.log.gz 2023-12-05T23:07:29.921 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65218.log 2023-12-05T23:07:29.922 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57232.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57232.log.gz 2023-12-05T23:07:29.923 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69195.log 2023-12-05T23:07:29.924 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65218.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65218.log.gz 2023-12-05T23:07:29.924 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41182.log 2023-12-05T23:07:29.925 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69195.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69195.log.gz 2023-12-05T23:07:29.925 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76969.log 2023-12-05T23:07:29.926 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41182.log: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41182.log.gz 2023-12-05T23:07:29.926 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71797.log 2023-12-05T23:07:29.927 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.76969.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76969.log.gz 2023-12-05T23:07:29.927 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62701.log 2023-12-05T23:07:29.928 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71797.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71797.log.gz 2023-12-05T23:07:29.928 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65049.log 2023-12-05T23:07:29.930 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62701.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62701.log.gz 2023-12-05T23:07:29.931 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81997.log 2023-12-05T23:07:29.933 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65049.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65049.log.gz 2023-12-05T23:07:29.934 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64913.log 2023-12-05T23:07:29.934 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81997.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81997.log.gz 2023-12-05T23:07:29.934 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78461.log 2023-12-05T23:07:29.935 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64913.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64913.log.gz 2023-12-05T23:07:29.935 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74154.log 2023-12-05T23:07:29.936 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78461.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78461.log.gz 2023-12-05T23:07:29.936 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74811.log 2023-12-05T23:07:29.937 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74154.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74154.log.gz 2023-12-05T23:07:29.937 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35106.log 2023-12-05T23:07:29.938 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74811.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74811.log.gz 2023-12-05T23:07:29.938 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59466.log 2023-12-05T23:07:29.939 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.35106.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35106.log.gz 2023-12-05T23:07:29.939 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76498.log 2023-12-05T23:07:29.940 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59466.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59466.log.gz 2023-12-05T23:07:29.940 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60456.log 2023-12-05T23:07:29.942 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76498.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76498.log.gz 2023-12-05T23:07:29.943 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58119.log 2023-12-05T23:07:29.943 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60456.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60456.log.gz 2023-12-05T23:07:29.944 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47813.log 2023-12-05T23:07:29.945 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58119.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58119.log.gz 2023-12-05T23:07:29.945 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63029.log 2023-12-05T23:07:29.951 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47813.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47813.log.gz 2023-12-05T23:07:29.952 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46840.log 2023-12-05T23:07:29.953 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63029.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63029.log.gz 2023-12-05T23:07:29.953 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77680.log 2023-12-05T23:07:29.957 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46840.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49990.log.gz 2023-12-05T23:07:29.959 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46840.log.gz 2023-12-05T23:07:29.959 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71280.log 2023-12-05T23:07:29.960 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42827.log 2023-12-05T23:07:29.960 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77680.log: /var/log/ceph/cluster1-client.mirror.71280.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77680.log.gz 2023-12-05T23:07:29.960 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75856.log 2023-12-05T23:07:29.961 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71280.log.gz 2023-12-05T23:07:29.976 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75963.log 2023-12-05T23:07:29.977 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75856.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75856.log.gz 2023-12-05T23:07:29.977 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58764.log 2023-12-05T23:07:29.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75963.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52568.log.gz 2023-12-05T23:07:29.978 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78591.log 2023-12-05T23:07:29.978 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75963.log.gz 2023-12-05T23:07:29.980 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58764.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58764.log.gz 2023-12-05T23:07:29.981 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73391.log 2023-12-05T23:07:29.982 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54416.log 2023-12-05T23:07:29.982 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78591.log.gz 2023-12-05T23:07:29.982 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.48656.log 2023-12-05T23:07:29.982 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54416.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73391.log.gz 2023-12-05T23:07:29.983 INFO:teuthology.orchestra.run.smithi190.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54416.log.gz 2023-12-05T23:07:29.983 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71932.log 2023-12-05T23:07:29.990 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.0.48656.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.48656.log.gz 2023-12-05T23:07:29.991 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80177.log 2023-12-05T23:07:29.992 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80177.log: 26.5% -- replaced with /var/log/ceph/cluster2-client.admin.80177.log.gz 2023-12-05T23:07:29.996 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80931.log 2023-12-05T23:07:29.997 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61921.log 2023-12-05T23:07:29.997 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71932.log.gz 2023-12-05T23:07:29.997 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80931.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80931.log.gz 2023-12-05T23:07:29.997 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69975.log 2023-12-05T23:07:29.998 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63692.log 2023-12-05T23:07:29.998 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.69975.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69975.log.gz 2023-12-05T23:07:29.998 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76342.log 2023-12-05T23:07:29.998 INFO:teuthology.orchestra.run.smithi190.stderr: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61921.log.gz 2023-12-05T23:07:29.999 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63692.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63692.log.gz 2023-12-05T23:07:29.999 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66663.log 2023-12-05T23:07:29.999 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54962.log 2023-12-05T23:07:30.002 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76342.log: /var/log/ceph/cluster2-client.mirror.66663.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66663.log.gz 2023-12-05T23:07:30.008 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57902.log 2023-12-05T23:07:30.009 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54962.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54962.log.gz 2023-12-05T23:07:30.009 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57945.log 2023-12-05T23:07:30.011 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57902.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57902.log.gz 2023-12-05T23:07:30.011 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66954.log 2023-12-05T23:07:30.013 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57945.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57945.log.gz 2023-12-05T23:07:30.013 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55434.log 2023-12-05T23:07:30.014 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66954.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66954.log.gz 2023-12-05T23:07:30.014 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79579.log 2023-12-05T23:07:30.015 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55434.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55434.log.gz 2023-12-05T23:07:30.015 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56864.log 2023-12-05T23:07:30.016 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79579.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79579.log.gz 2023-12-05T23:07:30.016 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45296.log 2023-12-05T23:07:30.017 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56864.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56864.log.gz 2023-12-05T23:07:30.017 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64134.log 2023-12-05T23:07:30.023 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45296.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45296.log.gz 2023-12-05T23:07:30.024 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71730.log 2023-12-05T23:07:30.025 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64134.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64134.log.gz 2023-12-05T23:07:30.025 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64463.log 2023-12-05T23:07:30.026 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71730.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71730.log.gz 2023-12-05T23:07:30.027 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52145.log 2023-12-05T23:07:30.028 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64463.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64463.log.gz 2023-12-05T23:07:30.028 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73626.log 2023-12-05T23:07:30.029 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52145.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52145.log.gz 2023-12-05T23:07:30.029 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60673.log 2023-12-05T23:07:30.030 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73626.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73626.log.gz 2023-12-05T23:07:30.030 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63738.log 2023-12-05T23:07:30.031 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60673.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60673.log.gz 2023-12-05T23:07:30.032 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45753.log 2023-12-05T23:07:30.033 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63738.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63738.log.gz 2023-12-05T23:07:30.033 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54851.log 2023-12-05T23:07:30.034 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45753.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45753.log.gz 2023-12-05T23:07:30.034 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44368.log 2023-12-05T23:07:30.040 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54851.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54851.log.gz 2023-12-05T23:07:30.040 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81815.log 2023-12-05T23:07:30.040 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44368.log: 82.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44368.log.gz 2023-12-05T23:07:30.040 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79319.log 2023-12-05T23:07:30.040 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81815.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81815.log.gz 2023-12-05T23:07:30.041 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52884.log.gz 2023-12-05T23:07:30.041 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56434.log 2023-12-05T23:07:30.041 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67740.log 2023-12-05T23:07:30.041 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79319.log: /var/log/ceph/cluster2-client.mirror.56434.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79319.log.gz 2023-12-05T23:07:30.041 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65431.log 2023-12-05T23:07:30.042 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67740.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67740.log.gz 2023-12-05T23:07:30.042 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39437.log 2023-12-05T23:07:30.042 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56434.log.gz 2023-12-05T23:07:30.042 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75699.log 2023-12-05T23:07:30.043 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39437.log: 87.3% 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65431.log.gz 2023-12-05T23:07:30.043 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.39437.log.gz 2023-12-05T23:07:30.044 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62083.log 2023-12-05T23:07:30.044 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59420.log 2023-12-05T23:07:30.045 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75699.log: /var/log/ceph/cluster1-client.mirror.62083.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75699.log.gz 2023-12-05T23:07:30.045 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62083.log.gz 2023-12-05T23:07:30.045 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66002.log 2023-12-05T23:07:30.045 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68906.log 2023-12-05T23:07:30.046 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59420.log: /var/log/ceph/cluster1-client.mirror.66002.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59420.log.gz 2023-12-05T23:07:30.046 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66002.log.gz 2023-12-05T23:07:30.046 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75942.log 2023-12-05T23:07:30.046 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61853.log 2023-12-05T23:07:30.047 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68906.log: /var/log/ceph/cluster2-client.mirror.75942.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68906.log.gz 2023-12-05T23:07:30.047 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75942.log.gz 2023-12-05T23:07:30.047 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65606.log 2023-12-05T23:07:30.047 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39748.log 2023-12-05T23:07:30.048 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61853.log: /var/log/ceph/cluster2-client.mirror.65606.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65606.log.gz 2023-12-05T23:07:30.048 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61853.log.gz 2023-12-05T23:07:30.048 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79397.log 2023-12-05T23:07:30.049 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78643.log 2023-12-05T23:07:30.049 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39748.log: /var/log/ceph/cluster2-client.admin.79397.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39748.log.gz 2023-12-05T23:07:30.049 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79397.log.gz 2023-12-05T23:07:30.050 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76725.log 2023-12-05T23:07:30.051 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61328.log 2023-12-05T23:07:30.051 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78643.log: /var/log/ceph/cluster1-client.mirror.76725.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78643.log.gz 2023-12-05T23:07:30.051 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76725.log.gz 2023-12-05T23:07:30.051 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61713.log 2023-12-05T23:07:30.051 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82309.log 2023-12-05T23:07:30.052 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61328.log: /var/log/ceph/cluster1-client.mirror.61713.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61328.log.gz 2023-12-05T23:07:30.052 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61713.log.gz 2023-12-05T23:07:30.052 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80489.log 2023-12-05T23:07:30.052 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71055.log 2023-12-05T23:07:30.052 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82309.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82309.log.gz 2023-12-05T23:07:30.054 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80489.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70852.log 2023-12-05T23:07:30.054 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.80489.log.gz 2023-12-05T23:07:30.054 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71055.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71055.log.gz 2023-12-05T23:07:30.054 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.73513.log 2023-12-05T23:07:30.054 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42676.log 2023-12-05T23:07:30.055 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70852.log: /var/log/ceph/cluster1-client.mirror.1.73513.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70852.log.gz 2023-12-05T23:07:30.055 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75390.log 2023-12-05T23:07:30.055 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42676.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42676.log.gz 2023-12-05T23:07:30.056 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44088.log 2023-12-05T23:07:30.059 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75390.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75390.log.gz 2023-12-05T23:07:30.059 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.73513.log.gz 2023-12-05T23:07:30.059 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75600.log 2023-12-05T23:07:30.059 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44088.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44088.log.gz 2023-12-05T23:07:30.060 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59864.log 2023-12-05T23:07:30.060 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42440.log 2023-12-05T23:07:30.060 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75600.log: /var/log/ceph/cluster2-client.mirror.59864.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75600.log.gz 2023-12-05T23:07:30.061 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59535.log 2023-12-05T23:07:30.062 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42440.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59864.log.gz 2023-12-05T23:07:30.076 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78903.log 2023-12-05T23:07:30.077 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59535.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59535.log.gz 2023-12-05T23:07:30.078 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56390.log 2023-12-05T23:07:30.078 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78903.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78903.log.gz 2023-12-05T23:07:30.079 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.56692.log 2023-12-05T23:07:30.079 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56390.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56390.log.gz 2023-12-05T23:07:30.080 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43548.log 2023-12-05T23:07:30.096 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.3.56692.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74932.log.gz 2023-12-05T23:07:30.096 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82543.log 2023-12-05T23:07:30.097 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.43548.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43548.log.gz 2023-12-05T23:07:30.097 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50281.log 2023-12-05T23:07:30.098 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82543.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82543.log.gz 2023-12-05T23:07:30.098 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77888.log 2023-12-05T23:07:30.099 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50281.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50281.log.gz 2023-12-05T23:07:30.100 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70267.log 2023-12-05T23:07:30.100 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77888.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77888.log.gz 2023-12-05T23:07:30.101 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63277.log 2023-12-05T23:07:30.101 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70267.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70267.log.gz 2023-12-05T23:07:30.102 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45684.log 2023-12-05T23:07:30.103 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63277.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63277.log.gz 2023-12-05T23:07:30.103 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79683.log 2023-12-05T23:07:30.104 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45684.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45684.log.gz 2023-12-05T23:07:30.104 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49272.log 2023-12-05T23:07:30.105 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79683.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79683.log.gz 2023-12-05T23:07:30.105 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55979.log 2023-12-05T23:07:30.107 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49272.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49272.log.gz 2023-12-05T23:07:30.107 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49583.log 2023-12-05T23:07:30.108 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55979.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55979.log.gz 2023-12-05T23:07:30.108 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76200.log 2023-12-05T23:07:30.120 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44673.log 2023-12-05T23:07:30.121 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76200.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76200.log.gz 2023-12-05T23:07:30.152 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51706.log 2023-12-05T23:07:30.153 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44673.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44673.log.gz 2023-12-05T23:07:30.156 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49753.log.gz 2023-12-05T23:07:30.160 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60742.log 2023-12-05T23:07:30.161 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51706.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51706.log.gz 2023-12-05T23:07:30.162 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83220.log 2023-12-05T23:07:30.162 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60742.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60742.log.gz 2023-12-05T23:07:30.163 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.42849.log 2023-12-05T23:07:30.163 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83220.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83220.log.gz 2023-12-05T23:07:30.163 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67616.log 2023-12-05T23:07:30.166 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.1.42849.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.56692.log.gz 2023-12-05T23:07:30.166 INFO:teuthology.orchestra.run.smithi190.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.42849.log.gz 2023-12-05T23:07:30.167 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58471.log 2023-12-05T23:07:30.167 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42630.log 2023-12-05T23:07:30.168 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58471.log: 89.5% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58471.log.gz 2023-12-05T23:07:30.168 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67616.log.gz 2023-12-05T23:07:30.169 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78773.log 2023-12-05T23:07:30.169 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75213.log 2023-12-05T23:07:30.169 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78773.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42630.log.gz 2023-12-05T23:07:30.170 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78773.log.gz 2023-12-05T23:07:30.170 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57011.log 2023-12-05T23:07:30.170 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.40248.log 2023-12-05T23:07:30.171 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57011.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75213.log.gz 2023-12-05T23:07:30.171 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58700.log 2023-12-05T23:07:30.171 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.3.40248.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57011.log.gz 2023-12-05T23:07:30.174 INFO:teuthology.orchestra.run.smithi190.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.40248.log.gz 2023-12-05T23:07:30.175 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53682.log 2023-12-05T23:07:30.175 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52906.log 2023-12-05T23:07:30.175 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53682.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58700.log.gz 2023-12-05T23:07:30.175 INFO:teuthology.orchestra.run.smithi190.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53682.log.gz 2023-12-05T23:07:30.176 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46457.log 2023-12-05T23:07:30.176 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47532.log 2023-12-05T23:07:30.177 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46457.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46457.log.gz 2023-12-05T23:07:30.177 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42827.log.gz 2023-12-05T23:07:30.178 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81477.log 2023-12-05T23:07:30.178 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70380.log 2023-12-05T23:07:30.178 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81477.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81477.log.gz 2023-12-05T23:07:30.183 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47532.log.gz 2023-12-05T23:07:30.184 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65956.log 2023-12-05T23:07:30.184 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56367.log 2023-12-05T23:07:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65956.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70380.log.gz 2023-12-05T23:07:30.185 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46638.log 2023-12-05T23:07:30.185 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65956.log.gz 2023-12-05T23:07:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56367.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56367.log.gz 2023-12-05T23:07:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49314.log 2023-12-05T23:07:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40602.log 2023-12-05T23:07:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49314.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46638.log.gz 2023-12-05T23:07:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64371.log 2023-12-05T23:07:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40602.log: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.40602.log.gz 2023-12-05T23:07:30.188 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47939.log 2023-12-05T23:07:30.188 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64371.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64371.log.gz 2023-12-05T23:07:30.189 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42309.log 2023-12-05T23:07:30.190 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47939.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47939.log.gz 2023-12-05T23:07:30.190 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82752.log 2023-12-05T23:07:30.191 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42309.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42309.log.gz 2023-12-05T23:07:30.191 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64394.log 2023-12-05T23:07:30.192 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82752.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82752.log.gz 2023-12-05T23:07:30.192 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49425.log 2023-12-05T23:07:30.193 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.64394.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64394.log.gz 2023-12-05T23:07:30.193 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77940.log 2023-12-05T23:07:30.194 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49425.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49425.log.gz 2023-12-05T23:07:30.194 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74790.log 2023-12-05T23:07:30.195 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77940.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77940.log.gz 2023-12-05T23:07:30.195 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46684.log 2023-12-05T23:07:30.196 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74790.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74790.log.gz 2023-12-05T23:07:30.196 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50435.log 2023-12-05T23:07:30.197 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46684.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46684.log.gz 2023-12-05T23:07:30.197 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45797.log 2023-12-05T23:07:30.198 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50435.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50435.log.gz 2023-12-05T23:07:30.198 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59443.log 2023-12-05T23:07:30.204 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45797.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45797.log.gz 2023-12-05T23:07:30.204 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78747.log 2023-12-05T23:07:30.205 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59443.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59443.log.gz 2023-12-05T23:07:30.206 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68112.log 2023-12-05T23:07:30.206 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78747.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78747.log.gz 2023-12-05T23:07:30.207 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57922.log 2023-12-05T23:07:30.208 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68112.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68112.log.gz 2023-12-05T23:07:30.208 INFO:teuthology.orchestra.run.smithi190.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52906.log.gz 2023-12-05T23:07:30.208 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50652.log 2023-12-05T23:07:30.209 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80359.log 2023-12-05T23:07:30.209 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57922.log.gz 2023-12-05T23:07:30.210 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48457.log 2023-12-05T23:07:30.210 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80359.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80359.log.gz 2023-12-05T23:07:30.210 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68795.log 2023-12-05T23:07:30.211 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.48457.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48457.log.gz 2023-12-05T23:07:30.211 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59728.log 2023-12-05T23:07:30.212 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.68795.log: 84.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68795.log.gz 2023-12-05T23:07:30.212 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68381.log 2023-12-05T23:07:30.213 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59728.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59728.log.gz 2023-12-05T23:07:30.213 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69820.log 2023-12-05T23:07:30.214 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68381.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68381.log.gz 2023-12-05T23:07:30.214 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69174.log 2023-12-05T23:07:30.215 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.69820.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69820.log.gz 2023-12-05T23:07:30.215 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62789.log 2023-12-05T23:07:30.218 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69174.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69174.log.gz 2023-12-05T23:07:30.218 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62479.log 2023-12-05T23:07:30.219 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62789.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62789.log.gz 2023-12-05T23:07:30.219 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61397.log 2023-12-05T23:07:30.221 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62479.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62479.log.gz 2023-12-05T23:07:30.221 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51037.log 2023-12-05T23:07:30.222 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61397.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61397.log.gz 2023-12-05T23:07:30.222 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82127.log 2023-12-05T23:07:30.223 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51037.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51037.log.gz 2023-12-05T23:07:30.224 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59287.log 2023-12-05T23:07:30.224 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82127.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82127.log.gz 2023-12-05T23:07:30.225 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67885.log 2023-12-05T23:07:30.225 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59287.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59287.log.gz 2023-12-05T23:07:30.226 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67451.log 2023-12-05T23:07:30.226 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67885.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67885.log.gz 2023-12-05T23:07:30.227 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67098.log 2023-12-05T23:07:30.228 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67451.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67451.log.gz 2023-12-05T23:07:30.228 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41114.log 2023-12-05T23:07:30.231 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67098.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67098.log.gz 2023-12-05T23:07:30.231 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65304.log 2023-12-05T23:07:30.232 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41114.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41114.log.gz 2023-12-05T23:07:30.232 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69257.log 2023-12-05T23:07:30.233 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65304.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65304.log.gz 2023-12-05T23:07:30.234 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56177.log 2023-12-05T23:07:30.234 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69257.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69257.log.gz 2023-12-05T23:07:30.235 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48100.log 2023-12-05T23:07:30.236 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56177.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56177.log.gz 2023-12-05T23:07:30.236 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77394.log 2023-12-05T23:07:30.237 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48100.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48100.log.gz 2023-12-05T23:07:30.237 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68277.log 2023-12-05T23:07:30.238 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77394.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77394.log.gz 2023-12-05T23:07:30.239 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66280.log 2023-12-05T23:07:30.241 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68277.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68277.log.gz 2023-12-05T23:07:30.241 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44157.log 2023-12-05T23:07:30.243 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66280.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66280.log.gz 2023-12-05T23:07:30.246 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81503.log 2023-12-05T23:07:30.246 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44157.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44157.log.gz 2023-12-05T23:07:30.246 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60348.log 2023-12-05T23:07:30.246 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81503.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81503.log.gz 2023-12-05T23:07:30.247 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39930.log 2023-12-05T23:07:30.247 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60348.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60348.log.gz 2023-12-05T23:07:30.248 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77030.log 2023-12-05T23:07:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39930.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39930.log.gz 2023-12-05T23:07:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60867.log 2023-12-05T23:07:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77030.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77030.log.gz 2023-12-05T23:07:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50551.log 2023-12-05T23:07:30.250 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60867.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60867.log.gz 2023-12-05T23:07:30.250 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79761.log 2023-12-05T23:07:30.251 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50551.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50551.log.gz 2023-12-05T23:07:30.251 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39752.log 2023-12-05T23:07:30.252 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79761.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79761.log.gz 2023-12-05T23:07:30.252 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46928.log 2023-12-05T23:07:30.253 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39752.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39752.log.gz 2023-12-05T23:07:30.253 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81711.log 2023-12-05T23:07:30.254 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46928.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46928.log.gz 2023-12-05T23:07:30.255 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43311.log 2023-12-05T23:07:30.256 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81711.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81711.log.gz 2023-12-05T23:07:30.256 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66933.log 2023-12-05T23:07:30.262 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43311.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43311.log.gz 2023-12-05T23:07:30.262 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78148.log 2023-12-05T23:07:30.263 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66933.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66933.log.gz 2023-12-05T23:07:30.263 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62944.log 2023-12-05T23:07:30.264 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78148.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78148.log.gz 2023-12-05T23:07:30.264 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77706.log 2023-12-05T23:07:30.266 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62944.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62944.log.gz 2023-12-05T23:07:30.266 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69753.log 2023-12-05T23:07:30.267 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77706.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77706.log.gz 2023-12-05T23:07:30.267 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81451.log 2023-12-05T23:07:30.268 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69753.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69753.log.gz 2023-12-05T23:07:30.269 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58985.log 2023-12-05T23:07:30.269 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81451.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81451.log.gz 2023-12-05T23:07:30.269 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61000.log 2023-12-05T23:07:30.271 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58985.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58985.log.gz 2023-12-05T23:07:30.272 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77290.log 2023-12-05T23:07:30.274 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61000.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61000.log.gz 2023-12-05T23:07:30.274 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74071.log 2023-12-05T23:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77290.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77290.log.gz 2023-12-05T23:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52763.log 2023-12-05T23:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74071.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74071.log.gz 2023-12-05T23:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73912.log 2023-12-05T23:07:30.277 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52763.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52763.log.gz 2023-12-05T23:07:30.278 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42699.log 2023-12-05T23:07:30.278 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.73912.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73912.log.gz 2023-12-05T23:07:30.278 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51099.log 2023-12-05T23:07:30.279 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42699.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42699.log.gz 2023-12-05T23:07:30.280 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44309.log 2023-12-05T23:07:30.280 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51099.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51099.log.gz 2023-12-05T23:07:30.281 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46374.log 2023-12-05T23:07:30.282 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44309.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44309.log.gz 2023-12-05T23:07:30.282 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71977.log 2023-12-05T23:07:30.283 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.46374.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46374.log.gz 2023-12-05T23:07:30.284 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40006.log 2023-12-05T23:07:30.285 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71977.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71977.log.gz 2023-12-05T23:07:30.285 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78122.log 2023-12-05T23:07:30.285 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.40006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40006.log.gz 2023-12-05T23:07:30.286 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51158.log 2023-12-05T23:07:30.286 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78122.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78122.log.gz 2023-12-05T23:07:30.286 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82726.log 2023-12-05T23:07:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66568.log 2023-12-05T23:07:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82726.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82726.log.gz 2023-12-05T23:07:30.309 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82075.log 2023-12-05T23:07:30.309 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66568.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72208.log.gz 2023-12-05T23:07:30.310 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66568.log.gz 2023-12-05T23:07:30.312 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81191.log 2023-12-05T23:07:30.313 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82075.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82075.log.gz 2023-12-05T23:07:30.313 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81113.log 2023-12-05T23:07:30.313 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81191.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81191.log.gz 2023-12-05T23:07:30.313 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69009.log 2023-12-05T23:07:30.313 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81113.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81113.log.gz 2023-12-05T23:07:30.313 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79787.log 2023-12-05T23:07:30.314 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69009.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69009.log.gz 2023-12-05T23:07:30.314 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60193.log 2023-12-05T23:07:30.314 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79787.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79787.log.gz 2023-12-05T23:07:30.315 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81555.log 2023-12-05T23:07:30.315 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60193.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60193.log.gz 2023-12-05T23:07:30.316 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71460.log 2023-12-05T23:07:30.316 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81555.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81555.log.gz 2023-12-05T23:07:30.316 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43270.log 2023-12-05T23:07:30.317 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71460.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71460.log.gz 2023-12-05T23:07:30.317 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39540.log 2023-12-05T23:07:30.318 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43270.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43270.log.gz 2023-12-05T23:07:30.318 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42396.log 2023-12-05T23:07:30.319 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39540.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39540.log.gz 2023-12-05T23:07:30.319 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63847.log 2023-12-05T23:07:30.325 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42396.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42396.log.gz 2023-12-05T23:07:30.325 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55933.log 2023-12-05T23:07:30.326 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63847.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63847.log.gz 2023-12-05T23:07:30.327 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82674.log 2023-12-05T23:07:30.328 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55933.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55933.log.gz 2023-12-05T23:07:30.328 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63897.log 2023-12-05T23:07:30.329 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82674.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82674.log.gz 2023-12-05T23:07:30.329 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75898.log 2023-12-05T23:07:30.330 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63897.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63897.log.gz 2023-12-05T23:07:30.331 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56321.log 2023-12-05T23:07:30.331 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.75898.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75898.log.gz 2023-12-05T23:07:30.333 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47747.log 2023-12-05T23:07:30.333 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56321.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56321.log.gz 2023-12-05T23:07:30.333 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72131.log 2023-12-05T23:07:30.338 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47747.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47747.log.gz 2023-12-05T23:07:30.339 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39826.log 2023-12-05T23:07:30.339 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.72131.log: 3.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72131.log.gz 2023-12-05T23:07:30.340 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58512.log 2023-12-05T23:07:30.340 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39826.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39826.log.gz 2023-12-05T23:07:30.340 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72045.log 2023-12-05T23:07:30.342 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58512.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58512.log.gz 2023-12-05T23:07:30.345 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42653.log 2023-12-05T23:07:30.346 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.72045.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72045.log.gz 2023-12-05T23:07:30.346 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60763.log 2023-12-05T23:07:30.346 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42653.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42653.log.gz 2023-12-05T23:07:30.346 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47297.log 2023-12-05T23:07:30.347 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60763.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60763.log.gz 2023-12-05T23:07:30.347 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63761.log 2023-12-05T23:07:30.353 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47297.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47297.log.gz 2023-12-05T23:07:30.353 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41238.log 2023-12-05T23:07:30.354 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63761.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63761.log.gz 2023-12-05T23:07:30.354 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60085.log 2023-12-05T23:07:30.356 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41238.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41238.log.gz 2023-12-05T23:07:30.356 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44224.log 2023-12-05T23:07:30.358 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60085.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60085.log.gz 2023-12-05T23:07:30.358 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71257.log 2023-12-05T23:07:30.364 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44224.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44224.log.gz 2023-12-05T23:07:30.365 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45996.log 2023-12-05T23:07:30.365 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71257.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71257.log.gz 2023-12-05T23:07:30.365 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79215.log 2023-12-05T23:07:30.367 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45996.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45996.log.gz 2023-12-05T23:07:30.368 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77628.log 2023-12-05T23:07:30.368 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79215.log.gz 2023-12-05T23:07:30.369 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53703.log 2023-12-05T23:07:30.369 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77628.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77628.log.gz 2023-12-05T23:07:30.369 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39830.log 2023-12-05T23:07:30.370 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.53703.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53703.log.gz 2023-12-05T23:07:30.370 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47168.log 2023-12-05T23:07:30.371 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39830.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39830.log.gz 2023-12-05T23:07:30.371 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79423.log 2023-12-05T23:07:30.377 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47168.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47168.log.gz 2023-12-05T23:07:30.378 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49462.log 2023-12-05T23:07:30.378 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79423.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79423.log.gz 2023-12-05T23:07:30.378 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66974.log 2023-12-05T23:07:30.379 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49462.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49462.log.gz 2023-12-05T23:07:30.379 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51438.log 2023-12-05T23:07:30.382 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66974.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66974.log.gz 2023-12-05T23:07:30.382 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74360.log 2023-12-05T23:07:30.383 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51438.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51438.log.gz 2023-12-05T23:07:30.383 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71640.log 2023-12-05T23:07:30.384 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74360.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74360.log.gz 2023-12-05T23:07:30.384 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35038.log 2023-12-05T23:07:30.385 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71640.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71640.log.gz 2023-12-05T23:07:30.386 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62369.log 2023-12-05T23:07:30.386 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.35038.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35038.log.gz 2023-12-05T23:07:30.386 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48925.log 2023-12-05T23:07:30.387 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62369.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62369.log.gz 2023-12-05T23:07:30.387 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68236.log 2023-12-05T23:07:30.388 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48925.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48925.log.gz 2023-12-05T23:07:30.389 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62152.log 2023-12-05T23:07:30.390 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68236.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68236.log.gz 2023-12-05T23:07:30.390 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55566.log 2023-12-05T23:07:30.391 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62152.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62152.log.gz 2023-12-05T23:07:30.392 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70200.log 2023-12-05T23:07:30.393 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55566.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55566.log.gz 2023-12-05T23:07:30.393 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67761.log 2023-12-05T23:07:30.394 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70200.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70200.log.gz 2023-12-05T23:07:30.394 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62835.log 2023-12-05T23:07:30.395 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67761.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67761.log.gz 2023-12-05T23:07:30.395 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81087.log 2023-12-05T23:07:30.396 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62835.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62835.log.gz 2023-12-05T23:07:30.397 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55480.log 2023-12-05T23:07:30.397 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81087.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81087.log.gz 2023-12-05T23:07:30.397 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55670.log 2023-12-05T23:07:30.398 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55480.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55480.log.gz 2023-12-05T23:07:30.398 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50348.log 2023-12-05T23:07:30.399 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55670.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55670.log.gz 2023-12-05T23:07:30.399 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63987.log 2023-12-05T23:07:30.400 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50348.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50348.log.gz 2023-12-05T23:07:30.400 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65362.log 2023-12-05T23:07:30.401 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63987.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63987.log.gz 2023-12-05T23:07:30.402 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59144.log 2023-12-05T23:07:30.402 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65362.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65362.log.gz 2023-12-05T23:07:30.403 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76242.log 2023-12-05T23:07:30.404 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.59144.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59144.log.gz 2023-12-05T23:07:30.404 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73811.log 2023-12-05T23:07:30.405 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76242.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76242.log.gz 2023-12-05T23:07:30.406 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63964.log 2023-12-05T23:07:30.406 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73811.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73811.log.gz 2023-12-05T23:07:30.407 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45449.log 2023-12-05T23:07:30.408 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63964.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63964.log.gz 2023-12-05T23:07:30.408 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61064.log 2023-12-05T23:07:30.409 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.45449.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45449.log.gz 2023-12-05T23:07:30.409 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66170.log 2023-12-05T23:07:30.410 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61064.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61064.log.gz 2023-12-05T23:07:30.410 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.52906.log 2023-12-05T23:07:30.411 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66170.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66170.log.gz 2023-12-05T23:07:30.411 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49541.log 2023-12-05T23:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.0.52906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55388.log 2023-12-05T23:07:30.429 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49541.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49541.log.gz 2023-12-05T23:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50039.log 2023-12-05T23:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55388.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55388.log.gz 2023-12-05T23:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42584.log 2023-12-05T23:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50039.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50039.log.gz 2023-12-05T23:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49314.log.gz 2023-12-05T23:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54810.log 2023-12-05T23:07:30.447 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42584.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42584.log.gz 2023-12-05T23:07:30.447 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.66324.log 2023-12-05T23:07:30.448 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54810.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54810.log.gz 2023-12-05T23:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48541.log 2023-12-05T23:07:30.468 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.2.66324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47274.log 2023-12-05T23:07:30.473 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48541.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42440.log.gz 2023-12-05T23:07:30.477 INFO:teuthology.orchestra.run.smithi190.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.52906.log.gz 2023-12-05T23:07:30.484 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58898.log 2023-12-05T23:07:30.485 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47274.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.66324.log.gz 2023-12-05T23:07:30.489 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76342.log.gz 2023-12-05T23:07:30.490 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47274.log.gz 2023-12-05T23:07:30.491 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.50674.log 2023-12-05T23:07:30.491 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47641.log 2023-12-05T23:07:30.492 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59210.log 2023-12-05T23:07:30.492 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58898.log.gz 2023-12-05T23:07:30.492 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.0.50674.log: /var/log/ceph/cluster2-client.mirror.47641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51517.log 2023-12-05T23:07:30.492 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59210.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59210.log.gz 2023-12-05T23:07:30.493 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45641.log 2023-12-05T23:07:30.493 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51517.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51517.log.gz 2023-12-05T23:07:30.494 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55716.log 2023-12-05T23:07:30.495 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45641.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45641.log.gz 2023-12-05T23:07:30.495 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71662.log 2023-12-05T23:07:30.496 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55716.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55716.log.gz 2023-12-05T23:07:30.496 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55132.log 2023-12-05T23:07:30.497 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71662.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71662.log.gz 2023-12-05T23:07:30.497 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44134.log 2023-12-05T23:07:30.498 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47641.log.gz 2023-12-05T23:07:30.498 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55132.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55132.log.gz 2023-12-05T23:07:30.499 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67843.log 2023-12-05T23:07:30.499 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66850.log 2023-12-05T23:07:30.500 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.67843.log: -- replaced with /var/log/ceph/cluster1-client.mirror.44134.log.gz 2023-12-05T23:07:30.501 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67843.log.gz 2023-12-05T23:07:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76825.log 2023-12-05T23:07:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54238.log 2023-12-05T23:07:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66850.log: /var/log/ceph/cluster1-client.mirror.76825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76825.log.gz 2023-12-05T23:07:30.505 INFO:teuthology.orchestra.run.smithi190.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66850.log.gz 2023-12-05T23:07:30.505 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80099.log 2023-12-05T23:07:30.506 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55624.log 2023-12-05T23:07:30.506 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54238.log: /var/log/ceph/cluster2-client.admin.80099.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80099.log.gz 2023-12-05T23:07:30.506 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54238.log.gz 2023-12-05T23:07:30.507 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51987.log 2023-12-05T23:07:30.507 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57801.log 2023-12-05T23:07:30.507 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55624.log: /var/log/ceph/cluster2-client.mirror.51987.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55624.log.gz 2023-12-05T23:07:30.508 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51576.log 2023-12-05T23:07:30.509 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57801.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57801.log.gz 2023-12-05T23:07:30.510 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53785.log 2023-12-05T23:07:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51576.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49583.log.gz 2023-12-05T23:07:30.513 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59512.log 2023-12-05T23:07:30.514 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53785.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53785.log.gz 2023-12-05T23:07:30.514 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75679.log 2023-12-05T23:07:30.515 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59512.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59512.log.gz 2023-12-05T23:07:30.516 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82049.log 2023-12-05T23:07:30.517 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75679.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75679.log.gz 2023-12-05T23:07:30.517 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61690.log 2023-12-05T23:07:30.518 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82049.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82049.log.gz 2023-12-05T23:07:30.518 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61022.log 2023-12-05T23:07:30.519 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.61690.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61690.log.gz 2023-12-05T23:07:30.519 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66524.log 2023-12-05T23:07:30.520 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.61022.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61022.log.gz 2023-12-05T23:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70290.log 2023-12-05T23:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66524.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66524.log.gz 2023-12-05T23:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83168.log 2023-12-05T23:07:30.538 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70290.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70290.log.gz 2023-12-05T23:07:30.538 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51805.log 2023-12-05T23:07:30.539 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83168.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83168.log.gz 2023-12-05T23:07:30.540 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82986.log 2023-12-05T23:07:30.541 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51805.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51805.log.gz 2023-12-05T23:07:30.541 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70492.log 2023-12-05T23:07:30.541 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82986.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82986.log.gz 2023-12-05T23:07:30.542 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58077.log 2023-12-05T23:07:30.543 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70492.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70492.log.gz 2023-12-05T23:07:30.543 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44696.log 2023-12-05T23:07:30.544 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58077.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58077.log.gz 2023-12-05T23:07:30.545 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65761.log 2023-12-05T23:07:30.546 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.44696.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44696.log.gz 2023-12-05T23:07:30.546 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78851.log 2023-12-05T23:07:30.547 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65761.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65761.log.gz 2023-12-05T23:07:30.548 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81243.log 2023-12-05T23:07:30.548 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78851.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78851.log.gz 2023-12-05T23:07:30.549 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80853.log 2023-12-05T23:07:30.549 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81243.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81243.log.gz 2023-12-05T23:07:30.550 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44824.log 2023-12-05T23:07:30.550 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80853.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80853.log.gz 2023-12-05T23:07:30.551 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73247.log 2023-12-05T23:07:30.564 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.44824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57186.log 2023-12-05T23:07:30.566 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73247.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73247.log.gz 2023-12-05T23:07:30.568 INFO:teuthology.orchestra.run.smithi190.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.50674.log.gz 2023-12-05T23:07:30.569 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77342.log 2023-12-05T23:07:30.570 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57186.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57186.log.gz 2023-12-05T23:07:30.570 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58921.log 2023-12-05T23:07:30.571 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77342.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77342.log.gz 2023-12-05T23:07:30.571 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54698.log 2023-12-05T23:07:30.572 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58921.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58921.log.gz 2023-12-05T23:07:30.573 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81893.log 2023-12-05T23:07:30.573 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.54698.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51158.log.gz 2023-12-05T23:07:30.574 INFO:teuthology.orchestra.run.smithi190.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54698.log.gz 2023-12-05T23:07:30.574 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77550.log 2023-12-05T23:07:30.575 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75428.log 2023-12-05T23:07:30.576 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81893.log: /var/log/ceph/cluster2-client.admin.77550.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81893.log.gz 2023-12-05T23:07:30.576 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77550.log.gz 2023-12-05T23:07:30.576 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.44368.log 2023-12-05T23:07:30.576 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80281.log 2023-12-05T23:07:30.576 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50652.log.gz 2023-12-05T23:07:30.577 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75428.log: /var/log/ceph/cluster2-client.mirror.1.44368.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75428.log.gz 2023-12-05T23:07:30.577 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68463.log 2023-12-05T23:07:30.578 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64267.log 2023-12-05T23:07:30.578 INFO:teuthology.orchestra.run.smithi190.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.44368.log.gz 2023-12-05T23:07:30.578 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80281.log: /var/log/ceph/cluster2-client.mirror.68463.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80281.log.gz 2023-12-05T23:07:30.579 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43247.log 2023-12-05T23:07:30.579 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65739.log 2023-12-05T23:07:30.580 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64267.log: /var/log/ceph/cluster1-client.mirror.43247.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43247.log.gz 2023-12-05T23:07:30.581 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68463.log.gz 2023-12-05T23:07:30.581 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64267.log.gz 2023-12-05T23:07:30.581 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50573.log 2023-12-05T23:07:30.581 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67864.log 2023-12-05T23:07:30.582 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65739.log: /var/log/ceph/cluster2-client.mirror.50573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81035.log 2023-12-05T23:07:30.583 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67864.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67864.log.gz 2023-12-05T23:07:30.583 INFO:teuthology.orchestra.run.smithi190.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65739.log.gz 2023-12-05T23:07:30.584 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56818.log 2023-12-05T23:07:30.584 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73312.log 2023-12-05T23:07:30.584 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81035.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81035.log.gz 2023-12-05T23:07:30.585 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.56818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39756.log 2023-12-05T23:07:30.585 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56818.log.gz 2023-12-05T23:07:30.585 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51965.log 2023-12-05T23:07:30.587 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39756.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73312.log.gz 2023-12-05T23:07:30.587 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39756.log.gz 2023-12-05T23:07:30.587 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62881.log 2023-12-05T23:07:30.587 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47790.log 2023-12-05T23:07:30.588 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51965.log: /var/log/ceph/cluster1-client.mirror.62881.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51965.log.gz 2023-12-05T23:07:30.588 INFO:teuthology.orchestra.run.smithi190.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62881.log.gz 2023-12-05T23:07:30.588 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80333.log 2023-12-05T23:07:30.593 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47790.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47790.log.gz 2023-12-05T23:07:30.594 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75514.log 2023-12-05T23:07:30.594 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57511.log 2023-12-05T23:07:30.594 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80333.log: /var/log/ceph/cluster1-client.mirror.75514.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80333.log.gz 2023-12-05T23:07:30.595 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76800.log 2023-12-05T23:07:30.595 INFO:teuthology.orchestra.run.smithi190.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75514.log.gz 2023-12-05T23:07:30.595 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57511.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57511.log.gz 2023-12-05T23:07:30.596 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45776.log 2023-12-05T23:07:30.596 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39678.log 2023-12-05T23:07:30.597 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76800.log.gz 2023-12-05T23:07:30.597 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67430.log 2023-12-05T23:07:30.598 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45776.log.gz 2023-12-05T23:07:30.598 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.39678.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39678.log.gz 2023-12-05T23:07:30.598 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50593.log 2023-12-05T23:07:30.598 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74890.log 2023-12-05T23:07:30.599 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50593.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50593.log.gz 2023-12-05T23:07:30.599 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67430.log.gz 2023-12-05T23:07:30.599 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78955.log 2023-12-05T23:07:30.600 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74890.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74890.log.gz 2023-12-05T23:07:30.600 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.75123.log 2023-12-05T23:07:30.601 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62986.log 2023-12-05T23:07:30.601 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78955.log: /var/log/ceph/cluster1-client.mirror.0.75123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78955.log.gz 2023-12-05T23:07:30.601 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53142.log 2023-12-05T23:07:30.602 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62986.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62986.log.gz 2023-12-05T23:07:30.603 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62280.log 2023-12-05T23:07:30.605 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53142.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.75123.log.gz 2023-12-05T23:07:30.616 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48499.log 2023-12-05T23:07:30.618 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62280.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62280.log.gz 2023-12-05T23:07:30.618 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60980.log 2023-12-05T23:07:30.619 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48499.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48499.log.gz 2023-12-05T23:07:30.620 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50674.log 2023-12-05T23:07:30.622 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60980.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60980.log.gz 2023-12-05T23:07:30.622 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52204.log 2023-12-05T23:07:30.626 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.50674.log: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50674.log.gz 2023-12-05T23:07:30.626 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45972.log 2023-12-05T23:07:30.628 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52204.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52204.log.gz 2023-12-05T23:07:30.629 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80437.log 2023-12-05T23:07:30.629 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.45972.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45972.log.gz 2023-12-05T23:07:30.630 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81321.log 2023-12-05T23:07:30.630 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80437.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80437.log.gz 2023-12-05T23:07:30.631 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41927.log 2023-12-05T23:07:30.631 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81321.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81321.log.gz 2023-12-05T23:07:30.632 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64309.log 2023-12-05T23:07:30.648 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.41927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76697.log 2023-12-05T23:07:30.650 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64309.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64309.log.gz 2023-12-05T23:07:30.664 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65477.log 2023-12-05T23:07:30.667 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76697.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76697.log.gz 2023-12-05T23:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42763.log 2023-12-05T23:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65477.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65477.log.gz 2023-12-05T23:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74601.log 2023-12-05T23:07:30.696 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.42763.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79891.log 2023-12-05T23:07:30.698 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74601.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74601.log.gz 2023-12-05T23:07:30.704 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65540.log 2023-12-05T23:07:30.704 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79891.log: 28.2% -- replaced with /var/log/ceph/cluster2-client.admin.79891.log.gz 2023-12-05T23:07:30.708 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42763.log.gz 2023-12-05T23:07:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67264.log 2023-12-05T23:07:30.717 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65540.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65540.log.gz 2023-12-05T23:07:30.732 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47999.log 2023-12-05T23:07:30.733 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67264.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67264.log.gz 2023-12-05T23:07:30.734 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51576.log.gz 2023-12-05T23:07:30.734 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57144.log 2023-12-05T23:07:30.735 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47999.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47999.log.gz 2023-12-05T23:07:30.736 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-12-05T23:07:30.738 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57144.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57144.log.gz 2023-12-05T23:07:30.738 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74092.log 2023-12-05T23:07:30.752 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63490.log 2023-12-05T23:07:30.753 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74092.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74092.log.gz 2023-12-05T23:07:30.768 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74659.log 2023-12-05T23:07:30.769 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44824.log.gz 2023-12-05T23:07:30.769 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62902.log 2023-12-05T23:07:30.769 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63490.log.gz 2023-12-05T23:07:30.770 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74659.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74659.log.gz 2023-12-05T23:07:30.770 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66809.log 2023-12-05T23:07:30.772 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62902.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62902.log.gz 2023-12-05T23:07:30.773 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75010.log 2023-12-05T23:07:30.774 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.66809.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66809.log.gz 2023-12-05T23:07:30.774 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81581.log 2023-12-05T23:07:30.776 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75010.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75010.log.gz 2023-12-05T23:07:30.776 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77966.log 2023-12-05T23:07:30.777 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81581.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81581.log.gz 2023-12-05T23:07:30.777 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82882.log 2023-12-05T23:07:30.778 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77966.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77966.log.gz 2023-12-05T23:07:30.778 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70920.log 2023-12-05T23:07:30.779 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82882.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82882.log.gz 2023-12-05T23:07:30.779 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40670.log 2023-12-05T23:07:30.780 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70920.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70920.log.gz 2023-12-05T23:07:30.780 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41063.log 2023-12-05T23:07:30.781 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40670.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40670.log.gz 2023-12-05T23:07:30.781 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71505.log 2023-12-05T23:07:30.782 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.41063.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.41063.log.gz 2023-12-05T23:07:30.782 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42494.log 2023-12-05T23:07:30.783 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71505.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71505.log.gz 2023-12-05T23:07:30.784 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40636.log 2023-12-05T23:07:30.784 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42494.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42494.log.gz 2023-12-05T23:07:30.785 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48908.log 2023-12-05T23:07:30.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.40636.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40636.log.gz 2023-12-05T23:07:30.786 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77238.log 2023-12-05T23:07:30.787 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48908.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48908.log.gz 2023-12-05T23:07:30.787 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45661.log 2023-12-05T23:07:30.788 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77238.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77238.log.gz 2023-12-05T23:07:30.788 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79475.log 2023-12-05T23:07:30.789 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45661.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45661.log.gz 2023-12-05T23:07:30.789 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.83116.log 2023-12-05T23:07:30.790 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79475.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79475.log.gz 2023-12-05T23:07:30.790 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81061.log 2023-12-05T23:07:30.791 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.83116.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.83116.log.gz 2023-12-05T23:07:30.791 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54524.log 2023-12-05T23:07:30.792 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81061.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81061.log.gz 2023-12-05T23:07:30.792 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82283.log 2023-12-05T23:07:30.793 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54524.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54524.log.gz 2023-12-05T23:07:30.793 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76112.log 2023-12-05T23:07:30.794 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82283.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82283.log.gz 2023-12-05T23:07:30.794 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78539.log 2023-12-05T23:07:30.795 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.76112.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76112.log.gz 2023-12-05T23:07:30.796 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66216.log 2023-12-05T23:07:30.796 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.78539.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78539.log.gz 2023-12-05T23:07:30.797 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57580.log 2023-12-05T23:07:30.797 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66216.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66216.log.gz 2023-12-05T23:07:30.798 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65262.log 2023-12-05T23:07:30.799 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57580.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57580.log.gz 2023-12-05T23:07:30.799 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67967.log 2023-12-05T23:07:30.801 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65262.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65262.log.gz 2023-12-05T23:07:30.802 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50968.log 2023-12-05T23:07:30.805 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.67967.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67967.log.gz 2023-12-05T23:07:30.805 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63825.log 2023-12-05T23:07:30.806 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50968.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50968.log.gz 2023-12-05T23:07:30.806 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49711.log 2023-12-05T23:07:30.808 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63825.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63825.log.gz 2023-12-05T23:07:30.809 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63231.log 2023-12-05T23:07:30.810 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49711.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49711.log.gz 2023-12-05T23:07:30.810 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41080.log 2023-12-05T23:07:30.811 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63231.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63231.log.gz 2023-12-05T23:07:30.811 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60588.log 2023-12-05T23:07:30.812 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.41080.log: 67.4% -- replaced with /var/log/ceph/cluster2-client.admin.41080.log.gz 2023-12-05T23:07:30.812 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63409.log 2023-12-05T23:07:30.814 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60588.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60588.log.gz 2023-12-05T23:07:30.814 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65342.log 2023-12-05T23:07:30.815 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.63409.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63409.log.gz 2023-12-05T23:07:30.816 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48020.log 2023-12-05T23:07:30.817 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.65342.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65342.log.gz 2023-12-05T23:07:30.817 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80749.log 2023-12-05T23:07:30.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48020.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48020.log.gz 2023-12-05T23:07:30.819 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58875.log 2023-12-05T23:07:30.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80749.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80749.log.gz 2023-12-05T23:07:30.820 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53274.log 2023-12-05T23:07:30.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58875.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58875.log.gz 2023-12-05T23:07:30.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72166.log 2023-12-05T23:07:30.823 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53274.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53274.log.gz 2023-12-05T23:07:30.823 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56131.log 2023-12-05T23:07:30.825 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.72166.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72166.log.gz 2023-12-05T23:07:30.825 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58587.log 2023-12-05T23:07:30.826 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.56131.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56131.log.gz 2023-12-05T23:07:30.826 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40239.log 2023-12-05T23:07:30.827 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58587.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58587.log.gz 2023-12-05T23:07:30.828 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58534.log 2023-12-05T23:07:30.844 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.40239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60262.log 2023-12-05T23:07:30.846 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.58534.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58534.log.gz 2023-12-05T23:07:30.863 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48541.log.gz 2023-12-05T23:07:30.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52303.log 2023-12-05T23:07:30.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77914.log 2023-12-05T23:07:30.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60262.log: /var/log/ceph/cluster2-client.mirror.52303.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60262.log.gz 2023-12-05T23:07:30.864 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52303.log.gz 2023-12-05T23:07:30.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51497.log 2023-12-05T23:07:30.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.77914.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77914.log.gz 2023-12-05T23:07:30.864 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82205.log 2023-12-05T23:07:30.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57344.log 2023-12-05T23:07:30.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.82205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82205.log.gz 2023-12-05T23:07:30.892 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65698.log 2023-12-05T23:07:30.893 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.57344.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57344.log.gz 2023-12-05T23:07:30.896 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51987.log.gz 2023-12-05T23:07:30.896 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58231.log 2023-12-05T23:07:30.897 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.65698.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65698.log.gz 2023-12-05T23:07:30.897 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47426.log 2023-12-05T23:07:30.898 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58231.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58231.log.gz 2023-12-05T23:07:30.899 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42332.log 2023-12-05T23:07:30.905 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47426.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47426.log.gz 2023-12-05T23:07:30.905 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45233.log 2023-12-05T23:07:30.906 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42332.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42332.log.gz 2023-12-05T23:07:30.906 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45077.log 2023-12-05T23:07:30.912 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45233.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45233.log.gz 2023-12-05T23:07:30.912 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73333.log 2023-12-05T23:07:30.914 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.45077.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45077.log.gz 2023-12-05T23:07:30.914 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.59030.log 2023-12-05T23:07:30.915 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73333.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73333.log.gz 2023-12-05T23:07:30.915 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69907.log 2023-12-05T23:07:30.932 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.0.59030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71100.log 2023-12-05T23:07:30.933 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.69907.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69907.log.gz 2023-12-05T23:07:30.948 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68339.log 2023-12-05T23:07:30.949 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71100.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71100.log.gz 2023-12-05T23:07:30.972 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66025.log 2023-12-05T23:07:30.975 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68339.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68339.log.gz 2023-12-05T23:07:30.980 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50889.log 2023-12-05T23:07:30.981 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.66025.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66025.log.gz 2023-12-05T23:07:30.988 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42286.log 2023-12-05T23:07:30.989 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50889.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50889.log.gz 2023-12-05T23:07:30.990 INFO:teuthology.orchestra.run.smithi190.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40239.log.gz 2023-12-05T23:07:30.990 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52526.log 2023-12-05T23:07:30.991 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.42286.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42286.log.gz 2023-12-05T23:07:30.991 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58425.log 2023-12-05T23:07:30.992 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52526.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52526.log.gz 2023-12-05T23:07:30.993 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69691.log 2023-12-05T23:07:30.993 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58425.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58425.log.gz 2023-12-05T23:07:30.994 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49773.log 2023-12-05T23:07:30.995 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69691.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69691.log.gz 2023-12-05T23:07:30.995 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62769.log 2023-12-05T23:07:30.995 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49773.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49773.log.gz 2023-12-05T23:07:30.996 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79007.log 2023-12-05T23:07:30.997 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.62769.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62769.log.gz 2023-12-05T23:07:30.997 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51689.log 2023-12-05T23:07:30.998 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.79007.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79007.log.gz 2023-12-05T23:07:30.998 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53041.log 2023-12-05T23:07:30.999 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51689.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51689.log.gz 2023-12-05T23:07:30.999 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70245.log 2023-12-05T23:07:31.000 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53041.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53041.log.gz 2023-12-05T23:07:31.000 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47575.log 2023-12-05T23:07:31.000 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70245.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70245.log.gz 2023-12-05T23:07:31.001 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57709.log 2023-12-05T23:07:31.007 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47575.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47575.log.gz 2023-12-05T23:07:31.007 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47469.log 2023-12-05T23:07:31.007 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57709.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57709.log.gz 2023-12-05T23:07:31.008 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80203.log 2023-12-05T23:07:31.012 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47469.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.59030.log.gz 2023-12-05T23:07:31.013 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47469.log.gz 2023-12-05T23:07:31.014 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55503.log 2023-12-05T23:07:31.015 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46661.log 2023-12-05T23:07:31.015 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80203.log.gz 2023-12-05T23:07:31.015 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80073.log 2023-12-05T23:07:31.017 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.46661.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55503.log.gz 2023-12-05T23:07:31.017 INFO:teuthology.orchestra.run.smithi190.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46661.log.gz 2023-12-05T23:07:31.017 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74767.log 2023-12-05T23:07:31.017 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.80073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77446.log 2023-12-05T23:07:31.017 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80073.log.gz 2023-12-05T23:07:31.020 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.74767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63536.log 2023-12-05T23:07:31.020 INFO:teuthology.orchestra.run.smithi190.stderr: 86.8%/var/log/ceph/cluster2-client.admin.77446.log: -- replaced with /var/log/ceph/cluster1-client.mirror.74767.log.gz 2023-12-05T23:07:31.020 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77446.log.gz 2023-12-05T23:07:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64527.log 2023-12-05T23:07:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55308.log 2023-12-05T23:07:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.63536.log: /var/log/ceph/cluster1-client.mirror.64527.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63536.log.gz 2023-12-05T23:07:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62392.log 2023-12-05T23:07:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.55308.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64527.log.gz 2023-12-05T23:07:31.022 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55308.log.gz 2023-12-05T23:07:31.022 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48567.log 2023-12-05T23:07:31.022 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-12-05T23:07:31.023 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.62392.log: /var/log/ceph/cluster2-client.mirror.48567.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48567.log.gz 2023-12-05T23:07:31.023 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62392.log.gz 2023-12-05T23:07:31.023 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43456.log 2023-12-05T23:07:31.023 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75123.log 2023-12-05T23:07:31.024 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-osd.2.log: /var/log/ceph/cluster1-client.mirror.43456.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43456.log.gz 2023-12-05T23:07:31.036 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60525.log 2023-12-05T23:07:31.037 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75123.log: 86.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75123.log.gz 2023-12-05T23:07:31.037 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52546.log 2023-12-05T23:07:31.038 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.60525.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60525.log.gz 2023-12-05T23:07:31.039 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76300.log 2023-12-05T23:07:31.039 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.52546.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52546.log.gz 2023-12-05T23:07:31.040 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58402.log 2023-12-05T23:07:31.040 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.76300.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50573.log.gz 2023-12-05T23:07:31.041 INFO:teuthology.orchestra.run.smithi190.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76300.log.gz 2023-12-05T23:07:31.041 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75535.log 2023-12-05T23:07:31.042 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.58402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48412.log 2023-12-05T23:07:31.042 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.75535.log: -- replaced with /var/log/ceph/cluster1-client.mirror.58402.log.gz 2023-12-05T23:07:31.042 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81633.log 2023-12-05T23:07:31.043 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.48412.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75535.log.gz 2023-12-05T23:07:31.044 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75662.log 2023-12-05T23:07:31.045 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.admin.81633.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81633.log.gz 2023-12-05T23:07:31.045 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51849.log 2023-12-05T23:07:31.051 INFO:teuthology.orchestra.run.smithi190.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48412.log.gz 2023-12-05T23:07:31.051 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.75662.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75662.log.gz 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59797.log 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.51849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64549.log 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51849.log.gz 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.59797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45525.log 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.64549.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59797.log.gz 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70447.log 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.0.45525.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64549.log.gz 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68988.log 2023-12-05T23:07:31.052 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.70447.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70447.log.gz 2023-12-05T23:07:31.053 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60650.log 2023-12-05T23:07:31.053 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68988.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.45525.log.gz 2023-12-05T23:07:31.053 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68174.log 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.60650.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60650.log.gz 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69050.log 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68988.log.gz 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.68174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50244.log 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69050.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68174.log.gz 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69050.log.gz 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53596.log 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.50244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73647.log 2023-12-05T23:07:31.056 INFO:teuthology.orchestra.run.smithi190.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50244.log.gz 2023-12-05T23:07:31.057 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57430.log 2023-12-05T23:07:31.057 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53596.log.gz 2023-12-05T23:07:31.057 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.73647.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73647.log.gz 2023-12-05T23:07:31.057 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74873.log 2023-12-05T23:07:31.058 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.57430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55604.log 2023-12-05T23:07:31.058 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.74873.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74873.log.gz 2023-12-05T23:07:31.059 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49948.log 2023-12-05T23:07:31.059 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57430.log.gz 2023-12-05T23:07:31.060 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.55604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43571.log 2023-12-05T23:07:31.060 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49948.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55604.log.gz 2023-12-05T23:07:31.072 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49948.log.gz 2023-12-05T23:07:31.072 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53063.log 2023-12-05T23:07:31.072 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.43571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71482.log 2023-12-05T23:07:31.072 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.53063.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43571.log.gz 2023-12-05T23:07:31.072 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47704.log 2023-12-05T23:07:31.072 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53142.log.gz 2023-12-05T23:07:31.072 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.71482.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71482.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74133.log 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69464.log 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.47704.log: /var/log/ceph/cluster2-client.mirror.74133.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51497.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74133.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54330.log 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.69464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42943.log 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.54330.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69464.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54330.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49091.log 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.admin.42943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41949.log 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.49091.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.42943.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49091.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47704.log.gz 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47125.log 2023-12-05T23:07:31.073 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster1-client.mirror.41949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52726.log 2023-12-05T23:07:31.074 INFO:teuthology.orchestra.run.smithi190.stderr: 74.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41949.log.gz 2023-12-05T23:07:31.075 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cluster2-client.mirror.47125.log: /var/log/ceph/cluster2-client.mirror.52726.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47125.log.gz 2023-12-05T23:07:31.163 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41927.log.gz 2023-12-05T23:07:31.248 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52726.log.gz 2023-12-05T23:07:31.269 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53063.log.gz 2023-12-05T23:07:35.117 INFO:teuthology.orchestra.run.smithi062.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-12-05T23:07:36.488 INFO:teuthology.orchestra.run.smithi062.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-12-05T23:07:36.713 INFO:teuthology.orchestra.run.smithi062.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-12-05T23:07:36.714 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-12-05T23:07:36.715 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m10.024s 2023-12-05T23:07:36.715 INFO:teuthology.orchestra.run.smithi062.stderr:user 0m28.939s 2023-12-05T23:07:36.715 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m1.026s 2023-12-05T23:08:14.396 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-12-05T23:08:16.562 INFO:teuthology.orchestra.run.smithi190.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-12-05T23:08:31.552 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-12-05T23:08:31.554 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-12-05T23:08:31.554 INFO:teuthology.orchestra.run.smithi190.stderr:real 1m4.838s 2023-12-05T23:08:31.554 INFO:teuthology.orchestra.run.smithi190.stderr:user 2m56.345s 2023-12-05T23:08:31.554 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m7.035s 2023-12-05T23:08:31.555 INFO:tasks.ceph:Archiving logs... 2023-12-05T23:08:31.556 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/remote/smithi062/log 2023-12-05T23:08:31.557 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-05T23:08:34.906 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/remote/smithi190/log 2023-12-05T23:08:34.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-05T23:08:56.976 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-05T23:08:56.994 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_4a50e96e7d9d85da28ef008617cb8342574bfd1b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-05T23:08:56.994 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-05T23:08:56.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-05T23:08:56.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-05T23:08:57.064 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-05T23:08:57.078 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-05T23:08:57.078 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-05T23:08:57.081 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-05T23:08:57.093 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2023-12-05T23:08:57.093 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2023-12-05T23:08:57.093 INFO:teuthology.orchestra.run.smithi062.stdout:*hv01.front.sepi 44.4.53.2 2 u 55 128 377 0.163 -4.588 0.485 2023-12-05T23:08:57.093 INFO:teuthology.orchestra.run.smithi062.stdout:+hv02.front.sepi 63.231.80.2 3 u 62 128 377 0.166 -2.223 0.827 2023-12-05T23:08:57.093 INFO:teuthology.orchestra.run.smithi062.stdout:+hv03.front.sepi 74.6.168.72 3 u 71 128 377 0.197 -3.129 0.518 2023-12-05T23:08:57.093 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-05T23:08:57.095 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2023-12-05T23:08:57.095 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2023-12-05T23:08:57.095 INFO:teuthology.orchestra.run.smithi190.stdout:*hv01.front.sepi 44.4.53.2 2 u 62 128 377 0.124 -10.569 1.877 2023-12-05T23:08:57.095 INFO:teuthology.orchestra.run.smithi190.stdout:+hv02.front.sepi 63.231.80.2 3 u 58 128 377 0.139 -7.843 1.947 2023-12-05T23:08:57.095 INFO:teuthology.orchestra.run.smithi190.stdout:+hv03.front.sepi 74.6.168.72 3 u 52 128 377 0.169 -8.011 1.578 2023-12-05T23:08:57.096 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-05T23:08:57.096 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-05T23:08:57.111 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-05T23:08:57.111 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-05T23:08:57.128 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-05T23:08:57.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-05T23:08:57.158 INFO:teuthology.task.internal:Duration was 3560.566024 seconds 2023-12-05T23:08:57.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-05T23:08:57.176 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-05T23:08:57.176 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-05T23:08:57.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-05T23:08:57.254 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-05T23:08:57.254 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2023-12-05T23:08:57.254 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-05T23:08:57.264 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2023-12-05T23:08:57.265 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-05T23:08:57.275 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-05T23:08:57.275 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-05T23:08:57.308 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-05T23:08:57.342 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-05T23:08:57.343 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-05T23:08:57.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-05T23:08:57.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-05T23:08:57.460 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-05T23:08:57.461 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-05T23:08:57.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-05T23:08:57.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-05T23:08:57.503 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-05T23:08:57.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-05T23:08:57.525 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2023-12-05T23:08:57.539 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2023-12-05T23:08:57.555 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-05T23:08:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T23:08:57.603 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-05T23:08:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-05T23:08:57.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-05T23:08:57.620 INFO:teuthology.task.internal:Transferring archived files... 2023-12-05T23:08:57.621 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/remote/smithi062 2023-12-05T23:08:57.621 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-05T23:08:57.664 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989/remote/smithi190 2023-12-05T23:08:57.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-05T23:08:57.682 INFO:teuthology.task.internal:Removing archive directory... 2023-12-05T23:08:57.682 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-05T23:08:57.704 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-05T23:08:57.724 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-05T23:08:57.741 INFO:teuthology.task.internal:Not uploading archives. 2023-12-05T23:08:57.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-05T23:08:57.758 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-05T23:08:57.759 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-05T23:08:57.761 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-05T23:08:57.765 INFO:teuthology.orchestra.run.smithi062.stdout: 658634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 5 23:08 /home/ubuntu/cephtest 2023-12-05T23:08:57.773 INFO:teuthology.orchestra.run.smithi190.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 5 23:08 /home/ubuntu/cephtest 2023-12-05T23:08:57.774 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-05T23:08:57.788 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-05T23:08:57.803 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi190 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-05T23:08:57.803 INFO:teuthology.nuke:Checking targets against current locks 2023-12-05T23:08:57.830 DEBUG:teuthology.nuke:shortname: smithi062 2023-12-05T23:08:57.830 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-05T23:08:57.850 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-05 22:01:04.971197', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKqroaZT7HdYX62YLD5DEPqGdvn/11IHy8kmg0TszQ4KFHbms/zi4XIFQ/5ysjy8atJOWH5R6XjS7bmPmFzLnsc='} 2023-12-05T23:08:57.853 DEBUG:teuthology.nuke:shortname: smithi190 2023-12-05T23:08:57.854 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-05T23:08:57.877 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-05_21:30:10-rbd-wip-dis-testing-distro-default-smithi/7478989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-05 22:01:04.972647', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWBgB238hKlGR5M5K13I6ZaYPHPvcklmz6GAjCQypqGTttuc6O4NwhdgZf/28VITHQriAuy2J0LPslmczqnfYo='} 2023-12-05T23:08:57.904 INFO:teuthology.orchestra.console.smithi062:Power off 2023-12-05T23:08:57.904 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-05T23:08:57.931 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2023-12-05T23:08:57.931 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-05T23:08:57.947 INFO:teuthology.orchestra.console.smithi190:Power off 2023-12-05T23:08:57.948 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-05T23:08:57.969 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2023-12-05T23:08:57.972 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2023-12-05T23:08:57.972 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-05T23:08:58.003 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-12-05T23:09:01.971 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-05T23:09:02.004 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-05T23:09:02.084 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2023-12-05T23:09:02.114 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-12-05T23:09:06.086 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-05T23:09:06.114 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-05T23:09:06.201 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2023-12-05T23:09:06.225 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2023-12-05T23:09:06.302 INFO:teuthology.orchestra.console.smithi062:Power off completed 2023-12-05T23:09:06.326 INFO:teuthology.orchestra.console.smithi190:Power off completed 2023-12-05T23:09:06.431 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 3560.566023826599 failure_reason: 'Command failed on smithi190 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json''' flavor: default owner: scheduled_dis@teuthology status: fail success: false 2023-12-05T23:09:06.431 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-05T23:09:06.516 INFO:teuthology.run:FAIL